Apoptosis, a process recently implicated as the cellular mechanism underlying ovarian follicular atresia and luteal regression, is characterized by the internucleosomal degradation of DNA by a Ca2+/Mg(2+)-dependent endonuclease. Although hormones and growth factors have been demonstrated to modulate the DNA degradation associated with ovarian follicular apoptosis, the nature and identity of the endonuclease involved is not known. Ca2+/Mg(2+)-dependent endonuclease activity has a developmental pattern of expression in rat granulosa and luteal cell nuclei. Thus, the present study was conducted to establish the presence of an endonuclease in the nuclei of ovarian granulosa and luteal cells and to examine the biochemical properties of the enzyme relevant to apoptosis. Nuclei from diethyl-stilbestrol (DES)-, eCG-, and hCG-primed rat ovaries were isolated and exposed to Ca2+ and Mg2+ in vitro. Nuclei from rat ovaries primed with eCG and hCG, but not DES, substantially degraded their DNA in an apoptotic fashion, and this DNA degradation was Ca2+/Mg(2+)-dependent and inhibited by Zn2+. Protein extracts from the nuclei of DES-, eCG-, and hCG-treated rat ovaries were tested for endonuclease activity by a plasmid degradation assay. The extracts were found to contain endonuclease activity with the same developmental pattern and cation dependency as found in intact nuclei. These protein extracts were assessed for nuclease activity by zymography, and three nuclease activities were identified depending on the type of DNA used in the gel and the electrophoresis conditions used for protein separation.(ABSTRACT TRUNCATED AT 250 WORDS)