Hindawi Publishing Corporation Boundary Value Problems Volume 2008, Article ID 217636, 10 pages doi:10.1155/2008/217636 Research Article Existence Result for a Class of Elliptic Systems with Indefinite Weights in R 2 Guoqing Zhang 1 and Sanyang Liu 2 1 College of Sciences, University of Shanghai for Science and Technology, Shanghai 200093, China 2 Department of Applied Mathematics, Xidian University, Xi’an 710071, China Correspondence should be addressed to Guoqing Zhang, zgqw2001@sina.com.cn Received 31 October 2007; Accepted 4 March 2008 Recommended by Zhitao Zhang We obtain the existence of a nontrivial solution for a class of subcritical elliptic systems with indefinite weights in R 2 . The proofs base on Trudinger-Moser inequality and a generalized linking theorem introduced by Kryszewski and Szulkin. Copyright q 2008 G. Zhang and S. Liu. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. 1. Introduction In this paper, we study the existence of a nontrivial solution for the following systems of two semilinear coupled Poisson equations P  −Δu  u  gx, v,x∈ R 2 , −Δv  v  fx, u,x∈ R 2 , 1.1 where fx, t and gx, t are continuous functions on R 2 × R and have the maximal growth on t which allows to treat problem P variationally, Δ is the Laplace operator. Recently, there exists an extensive bibliography in the study of elliptic problem in R N 1–6. As dimensions N ≥ 3, in 1998, de Figueiredo and Yang 5 considered the following coupled elliptic systems: −Δu  u  gx, v,x∈ R N , −Δv  v  fx, u,x∈ R N , 1.2 2 Boundary Value Problems where f, g are radially symmetric in x and satisfied the following Ambrosetti-Rabinowitz condition:  t 0 fx, sds ≥ c|t| 2δ 1 ,  t 0 gx, sds ≥ c|t| 2δ 2 , ∀t ∈ R, 1.3 and for some δ 1 > 0,δ 2 > 0. They obtained the decay, symmetry, and existence of solutions for problem 1.2. In 2004, Li and Yang 6 proved that problem 1.2 possesses at least a positive solution when the nonlinearities fx, t and gx, t are “asymptotically linear” at infinity and “superlinear” at zero, that is, 1 lim t→∞ fx, t/tl>1, lim t→∞ gx, t/tm>1, uniformly in x ∈ R N ; 2 lim t→0 fx, t/tlim t→0 gx, t/t0, uniformly with respect to x ∈ R N . In 2006, Colin and Frigon 1 studied the following systems of coupled Poission equations with critical growth in unbounded domains: −Δu  |v| 2 ∗ −2 v, −Δv  |u| 2 ∗ −2 u, 1.4 where 2 ∗  2N/N − 2 is critical Sobolev exponent, u, v ∈ D 1,2 0 Ω ∗  and Ω ∗  R N \ E with E   a∈Z N a  ω ∗ for a domain containing the origin ω ∗ ⊂ ω ∗ ⊂ B0, 1/2. Here, B0, 1/2 denotes the open ball centered at the origin of radius 1/2. The existence of a nontrivial solution was obtained by using a generalized linking theorem. As it is well known in dimensions N ≥ 3, the nonlinearities are required to have polynomial growth at infinity, so that one can define associated functionals in Sobolev spaces. Coming to dimension N  2, much faster growth is allowed for the nonlinearity. In fact, the Trudinger-Moser estimates in N  2 replace the Sobolev embedding theorem used in N ≥ 3. In dimension N  2, Adimurth and Yadava 7, de Figueiredo et al. 8 discussed the solvability of problems of the type −Δu  fx, u,x∈ Ω, u  0,x∈ ∂Ω, 1.5 where Ω is some bounded domain in R 2 . Shen et al. 9 considered the following nonlinear elliptic problems with critical potential: Δu − μ u  |x| log  R/|x|  2  fx, u,x∈ Ω u  0,x∈ ∂Ω, 1.6 and obtained some existence results. In the whole space R 2 , some authors considered the following single semilinear elliptic equations: −Δu  V xu  fx, u,x∈ R 2 . 1.7 G. Zhang and S. Liu 3 As the potential V x and the nonlinearity fx, t are asymptotic to a constant function, Cao 10 obtained the existence of a nontrivial solution. As the potential V x and the nonlinearity fx, t are asymptotically periodic at infinity, Alves et al. 11 proved the existence of at least one positive weak solution. Our aim in this paper is to establish the existence of a nontrivial solution for problem P in subcritical case. To our knowledge, there are no results in the literature establishing the existence of solutions to these problems in the whole space. However, it contains a basic difficulty. Namely, the energy functional associated with problem P has strong indefinite quadratic part, so there is not any more mountain pass structure but linking one. Therefore, the proofs of our main results cannot rely on classical min-max results. Combining a generalized linking theorem introduced by Kryszewski and Szulkin 12 and Trudinger-Moser inequality, we prove an existence result for problem P. The paper is organized as follows. In Section 2, we recall some results and state our main results. In Section 3, main result is proved. 2. Preliminaries and main results Consider the Hilbert space 13 H 1  R 2    u ∈ L 2  R 2  , ∇u ∈ L 2  R 2  , 2.1 and denote the product space Z  H 1 R 2  × H 1 R 2  endowed with the inner product:  u, v, φ, ψ    R 2 ∇u∇φ  uφdx   R 2 ∇v∇ψ  vψdx, ∀φ, ψ ∈ Z. 2.2 If we define Z   {u, u ∈ Z},Z −  {v,−v ∈ Z}. 2.3 It is easy to check that Z  Z  ⊕ Z − , since u, v 1 2 u  v, u  v 1 2 u − v, v − u. 2.4 Let us denote by P resp., Q the projection of Z on to Z  resp., Z − ,wehave 1 2    Pu, v   2 −   Qu, v   2   1 2     1 2 u  v, u  v     2 − 1 2     1 2 u − v, v − u     2  1 4   R 2  |∇u| 2  |∇v| 2  2∇u∇v  dx   R 2  |u| 2  |v| 2  2uv  dx −  R 2  |∇u| 2  |∇v| 2 − 2∇u∇v  dx −  R 2  |u| 2  |v| 2 − 2uv  dx    R 2 ∇u∇v  uvdx. 2.5 4 Boundary Value Problems Now, we define the functional Iu, v  R 2 ∇u∇v  uvdx −  R 2  Fx, uGx, v  dx    Pu, v   2 2 −   Qu, v   2 2 − ϕu, v, ∀u, v ∈ Z, 2.6 where ϕu, v  R 2  Fx, uGx, v  dx. 2.7 Let z 0 ∈ Z  \{0} and let R>r>0, we define M   z  z −  λz 0 : z − ∈ Z − , z≤R, λ ≥ 0  , M 0   z  z −  λz 0 : z − ∈ Z − , z  R and λ ≥ 0orz≤R and λ ≥ 0  , N   z ∈ Z  : z  r  . 2.8 Here, we assume the following condition: H1 f,g ∈ CR 2 × R, R; H2 lim t→0 fx, t/tlim t→0 gx, t/t0 uniformly with respect to x ∈ R 2 ; H3 there exist μ>2andη>0 such that 0 <μFx, t ≤ tfx, t, 0 <μGx, t ≤ tgx, t, ∀|t|≥η. 2.9 Lemma 2.1 see 12, 14. Assume (H1), (H2), and (H3), and suppose 1 Iz1/2Pz 2 −Qz 2 −ϕz, where ϕ ∈ C 1 Z, R is sequentially lower semicontinu- ous, bounded below, and ∇ϕ is weakly sequentially continuous; 2 there exist z 0 ∈ Z  \{0},α>0,andR>r>0, such that inf N Iz ≥ α>0, sup M 0 Iz ≤ 0. 2.10 Then, there exist c>0 and a sequence z n  ⊂ Z such that Iz n  −→ c, I  z n  −→ 0, as n −→ ∞ . 2.11 Moreover, c ≥ α. Theorem 2.2. Under the assumptions (H1), (H2), and (H3), if f and g has subcritical growth (see definition below), problem (P) possesses a nontrivial weak solution. G. Zhang and S. Liu 5 In the whole space R 2 , do ´ O and Souto 15 proved a version of Trudinger-Moser inequality, that is, i if u ∈ H 1 R 2 ,β>0, we have  R 2  exp  β|u| 2  − 1  dx < ∞; 2.12 ii if 0 <β<4π and |u| L 2 R 2  ≤ c, then there exists a constant c 2  c 1 c, β such that sup |∇u| L 2 R 2  ≤1  R 2  exp  β|u| 2  − 1  dx < c 2 . 2.13 Definition 2.3. We say fx, t has subcritical growth at ∞, if for all β>0, there exists a positive constant c 3 such that fx, t ≤ c 3 exp  βt 2  , ∀x, t ∈ R 2 × 0, ∞. 2.14 3. Proof of Theorem 2.2 In this section, we will prove Theorem 2.2. under our assumptions and 2.14, there exist c ε > 0,β > 0 such that   Fx, t   ,   Gx, t   ≤ t 2 2 ε  c ε  exp  βt 2  − 1  , ∀ε>0, ∀t ∈ R. 3.1 Then, we obtain Fx, u,Gx, v ∈ L 2  R 2  , ∀u, v ∈ H 1  R 2  . 3.2 Therefore, the functional Iu, v is well defined. Furthermore, using standard arguments, we obtain the functional Iu, v is C 1 functional in Z and I  u, vφ, ψ  R 2 ∇u∇ψ  uψdx   R 2  ∇v∇φ  vφ  dx −  R 2  fx, uφ  gx, vψ  dx, ∀φ, ψ ∈ Z. 3.3 Consequently, the weak solutions of problem P are exactly the critical points of Iu, v in Z. Now, we prove that the functional Iu, v satisfied the geometry of Lemma 2.1. Lemma 3.1. There exist r>0 and α>0 such that inf N Iu, u ≥ α>0. Proof. By 2.14 and assumption H2, there exists c ε > 0 such that Fx, t,Gx, t ≤ t 2 2 ε  c ε t 3  exp  βt 2  − 1  , ∀t ∈ R, 3.4 6 Boundary Value Problems and thus on N, we have Iu, u ≥  R 2  |∇u| 2  u 2  dx −  R 2  εu 2  c ε u 3  exp  βu 2  − 1  dx ≥  R 2  |∇u| 2  u 2  dx − ε  R 2 u 2 dx − c ε   R 2 u 6 dx  1/2   R 2  exp  βu 2  − 1  2 dx  1/2 ≥  R 2  |∇u| 2  u 2  dx − ε  R 2 u 2 dx − c ε u 3   R 2 exp  βu 2  − 1  dx  1/2 . 3.5 So, by the Sobolev embedding theorem and 2.12, we can choose r>0sufficiently small, such that Iu, u ≥ α>0, whenever u  r. 3.6 Lemma 3.2. There exist u 0 ,u 0  ∈ Z  \{0} and R>r>0 such that sup M 0 I ≤ 0. Proof. 1 By assumption H3,wehaveonZ − Iu, u  R 2  |∇u| 2  u 2  dx −  R 2  Fx, uGx, −u  dx ≤ 0 3.7 because Fx, t ≥ 0,Gx, t ≥ 0 for any x, t ∈ R 2 × R. 2 Assumption H3 implies that there exist c 4 > 0,c 5 > 0 such that Fx, t,Gx, t ≥ c 4 t μ − c 5 , ∀t ∈ R. 3.8 Now, we choose u 0 ,u 0  ∈ Z  \{0} such that u 0 ,u 0   r,then I  −v, vλ  u 0 ,u 0   λ 2  R 2  |∇u 0 | 2  u 2 0  dx −  R 2  |∇v| 2  v 2  dx −  R 2  Fλu 0  v   G  λu 0 − v  dx ≤−  R 2  |∇u| 2  u 2  dx  c  λ 2 − λ μ  . 3.9 Because μ>2, it follows that for w ∈ M 0 Iw −→ − ∞ , whenever w−→∞, 3.10 and so, taking R>rlarge, we get sup M 0 I ≤ 0. G. Zhang and S. Liu 7 Proof of Theorem 2.2. By Lemma 3.1, there exist r>0andα>0 such that inf N Iu, u ≥ α>0. By Lemma 3.2, there exist u 0 ,u 0  ∈ Z  \{0} and R>r>0 such that sup M 0 I ≤ 0. Since Z  Z  ⊕Z − , we have Iu, v  R 2 ∇u∇v  uvdx −  R 2  Fx, uGx, v  dx    Pu, v   2 2 −   Qu, v   2 2 − ϕu, v, ∀u, v ∈ Z. 3.11 From 2.14, 3.1, and assumption H3, ϕu, v ∈ C 1 ,ϕu, v ≥ 0andϕu, v is sequentially lower semicontinuous by Z ⊂ L 2 loc R 2 ×L 2 loc R 2  and Fatou’s lemma; ∇ϕ is weakly sequentially continuous. Thus, by Lemma 2.1 there exists a sequence u n ,v n  ⊂ Z such that Iu n ,v n  −→ c ≥ α, I  u n ,v n  −→ 0. 3.12 Claim 3.3. There is c<∞, such that u n ,v n ≤c for any n. Indeed, from 3.12,weobtain that the sequence u n ,v n  ⊂ Z satisfies I  u n ,v n   c  δ n ,I   u n ,v n  φ, ψε n    u n ,v n    , as n −→ ∞ , 3.13 where φ, ψ ∈{u n ,v n },δ n → 0,ε n → 0asn →∞. Taking φ, ψ{u n ,v n } in 3.13 and assumption H3,wehave  R 2  f  x, u n  u n  g  x, v n  v n  dx ≤ 2  R 2  F  x, u n   G  x, v n  dx  2c  2δ n  ε n   u n ,v n    ≤ 2 μ  R 2  fx, u n  u n  g  x, v n  v n  dx  C  2δ n  ε n   u n ,v n    , 3.14 where C depends only on c and η in assumption H3. Since μ>2, we have 1 − 2/μ > 0, and thus  1 − 2 μ   R 2  f  x, u n  u n  g  x, v n  v n  dx ≤ C  2δ n  ε n    u n ,v n    , ∀n ∈ N. 3.15 On the other hand, let φ, ψv n , 0, φ, ψ0,u n  in 3.13,weobtain   v n   2 − ε n   v n   ≤  R 2 f  x, u n  v n dx,   u n   2 − ε n   u n   ≤  R 2 g  x, v n  u n dx. 3.16 that is,   v n   ≤  R 2 f  x, u n  v n   v n   dx  ε n ,   u n   ≤  R 2 g  x, v n  u n   u n   dx  ε n . 3.17 8 Boundary Value Problems Now, we recall the following inequality see 7, Lemma 2.4 : mn ≤ ⎧ ⎪ ⎨ ⎪ ⎩  e n 2 − 1   mlog m 1/2 ,n≥ 0,m≥ e 1/4 ,  e n 2 − 1   1 2 m 2 ,n≥ 0, 0 ≤ m ≤ e 1/4 . 3.18 Let n  v n /v n  and m  fx, u n /c 3 , where c 3 is defined in 2.14,wehave c 3  R 2 f  x, u n  c 3 v n v n  dx ≤ c 3  R 2  exp  v n v n   2 − 1  dx  c 3  {x∈R 2 ,fx,u n /c 3 ≥e 1/4 } f  x, u n  c 3  log f  x, u n  c 3  1/2 dx  c 3  {x∈R 2 ,fx,u n /c 3 ≤e 1/4 }  f  x, u n  c 3  2 dx. 3.19 By 2.12,wehave  R 2 exp v n /v n  2 − 1dx < ∞. By 2.14,wehave  log fx, t c 3  1/2 ≤ β 1/2 t. 3.20 Hence, we have c 3  R 2 f  x, u n  c 3 v n   v n   dx ≤ c 6  β 1/2  R 2 f  x, u n  u n dx 3.21 for some positive constant c 6 . So we have   v n   ≤ c 6  β 1/2  R 2 f  x, u n  u n dx  ε n . 3.22 Using a similar argument, we obtain   u n   ≤ c 7  β 1/2  R 2 g  x, v n  v n dx  ε n 3.23 for some positive constant c 7 . Combining 3.22 and 3.23,wehave    u n ,v n    ≤ c 8  1  δ n  ε n    u n ,v n     ε n  3.24 for some positive constant c 8 , which implies that u n ,v n ≤c. Thus, for a subsequence still denoted by u n ,v n , there is u 0 ,v 0  ∈ Z such that  u n ,v n  −→  u 0 ,v 0  weakly in Z, as n −→ ∞ ,  u n ,v n  −→  u 0 ,v 0  in L s loc  R 2  × L s loc R 2  for s ≥ 1, as n −→ ∞ ,  u n x,v n x  −→  u 0 x,v 0 x  , almost every, in R 2 , as n −→ ∞ . 3.25 G. Zhang and S. Liu 9 Then, there exists hx ∈ H 1 R 2  such that |u n x|≤h, ∀x ∈ R 2 , ∀n ∈ N. From 2.12 and 2.14,wehave  R 2 expβh 2 x − 1dx < c, this implies  R 2 f  x, u n  φdx −→  R 2 f  x, u 0  φdx, as n −→ ∞ . 3.26 Similarly, we can obtain  R 2 g  x, v n  ψdx −→  R 2 g  x, v 0  ψdx, as n −→ ∞ . 3.27 From these, we have I  u n ,v n φ, ψ0, so u 0 ,v 0  is weak solution of problem P. Claim 3.4. u 0 ,v 0  is nontrivial. By contradiction, since fx, t has subcritical growth, from 2.14 and H ¨ older inequality, we have  R 2 f  x, u n  u n dx ≤ c  R 2 u n  exp  βu 2 n  − 1  dx ≤ c    R 2 |u n | q  dx  1/q    R 2  exp  βqu 2 n  − 1  dx  1/q , 3.28 where 1/q   1/q  1. Choosing suitable β and q, we have  R 2  exp  βqu 2 n  − 1  dx ≤ c. 3.29 Then, we obtain  R 2 f  x, u n  u n dx ≤ c   R 2   u n   q  dx  1/q  . 3.30 Since u n → 0inL q  R 2 , as n →∞, this will lead to  R 2 f  x, u n  u n dx −→ 0, as n −→ ∞ . 3.31 Similarly, we have  R 2 g  x, v n  v n dx −→ 0, as n −→ ∞ . 3.32 Using assumption H3,weobtain  R 2 F  x, u n  dx −→ 0,  R 2 G  x, v n  dx −→ 0, as n −→ ∞ . 3.33 This together with I  u n ,v n u n ,v n  → 0, we have  R 2  ∇u n ∇v n  u n v n  dx −→ 0, as n −→ ∞ . 3.34 Thus, we see that I  u n ,v n  −→ 0, as n −→ ∞ . 3.35 which is a contradiction to Iu n ,v n  → c ≥ α>0, as n →∞. Consequently, we have a nontrivial critical point of the functional Iu, v and conclude the proof of Theorem 2.2. 10 Boundary Value Problems Acknowledgment This work is supported by Innovation Program of Shanghai Municipal Education Commission under Grant no. 08 YZ93. References 1 F. Colin and M. Frigon, “Systems of coupled Poisson equations with critical growth in unbounded domains,” Nonlinear Differential Equations and Applications, vol. 13, no. 3, pp. 369–384, 2006. 2 Y. Ding and S. Li, “Existence of entire solutions for some elliptic systems,” Bulletin of the Australian Mathematical Society, vol. 50, no. 3, pp. 501–519, 1994. 3 D. G. de Figueiredo, “Nonlinear elliptic systems,” Anais da Academia Brasileira de Ci ˆ encias,vol.72,no.4, pp. 453–469, 2000. 4 D. G. de Figueiredo, J. M. do ´ O, and B. Ruf, “Critical and subcritical elliptic systems in dimension two,” Indiana University Mathematics Journal, vol. 53, no. 4, pp. 1037–1054, 2004. 5 D. G. de Figueiredo and J. Yang, “Decay, symmetry and existence of solutions of semilinear elliptic systems,” Nonlinear Analysis: Theory, Methods & Applications, vol. 33, no. 3, pp. 211–234, 1998. 6 G. Li and J. Yang, “Asymptotically linear elliptic systems,” Communications in Partial Differential Equations, vol. 29, no. 5-6, pp. 925–954, 2004. 7 Adimurthi and S. L. Yadava, “Multiplicity results for semilinear elliptic equations in a bounded domain of R 2 involving critical exponents,” Annali della Scuola Normale Superiore di Pisa, vol. 17, no. 4, pp. 481–504, 1990. 8 D. G. de Figueiredo, O. H. Miyagaki, and B. Ruf, “Elliptic equations in R 2 with nonlinearities in the critical growth range,” Calculus of Variations and Partial Differential Equations, vol. 3, no. 2, pp. 139–153, 1995. 9 Y. Shen, Y. Yao, and Z. Chen, “On a class of nonlinear elliptic problem with critical potential in R 2 ,” Science in China Series A, vol. 34, pp. 610–624, 2004. 10 D. M. Cao, “Nontrivial solution of semilinear elliptic equation with critical exponent in R 2 ,” Communications in Partial Differential Equations, vol. 17, no. 3-4, pp. 407–435, 1992. 11 C. O. Alves, J. M. do ´ O, and O. H. Miyagaki, “On nonlinear perturbations of a periodic elliptic problem in R 2 involving critical growth,” Nonlinear Analysis: Theory, Methods & Applications, vol. 56, no. 5, pp. 781–791, 2004. 12 W. Kryszewski and A. Szulkin, “Generalized linking theorem with an application to a semilinear Schr ¨ odinger equation,” Advances in Differential Equations, vol. 3, no. 3, pp. 441–472, 1998. 13 M. Willem, Minimax Theorems,vol.24ofProgress in Nonlinear Differential Equations and Their Applications, Birkh ¨ auser, Boston, Mass, USA, 1996. 14 G. Li and A. Szulkin, “An asymptotically periodic Schr ¨ odinger equation with indefinite linear part,” Communications in Contemporary Mathematics, vol. 4, no. 4, pp. 763–776, 2002. 15 J. M. do ´ O and M. A. S. Souto, “On a class of nonlinear Schr ¨ odinger equations in R 2 involving critical growth,” Journal of Differential Equations, vol. 174, no. 2, pp. 289–311, 2001. . existence of a nontrivial solution for a class of subcritical elliptic systems with indefinite weights in R 2 . The proofs base on Trudinger-Moser inequality and a generalized linking theorem introduced. the proofs of our main results cannot rely on classical min-max results. Combining a generalized linking theorem introduced by Kryszewski and Szulkin 12 and Trudinger-Moser inequality, we prove an. Hindawi Publishing Corporation Boundary Value Problems Volume 2008, Article ID 217636, 10 pages doi:10.1155/2008/217636 Research Article Existence Result for a Class of Elliptic Systems with Indefinite