This page (revision-31) was last changed on 07-Dec-2016 14:14 by HuiLi

This page was created on 18-Sep-2007 12:20 by JianSun

Only authorized users are allowed to rename pages.

Only authorized users are allowed to delete pages.

Page revision history

Version Date Modified Size Author Changes ... Change note
31 07-Dec-2016 14:14 4 KB HuiLi to previous
30 02-Oct-2007 14:39 4 KB HuiLi to previous | to last
29 02-Oct-2007 14:33 4 KB HuiLi to previous | to last
28 02-Oct-2007 14:28 4 KB HuiLi to previous | to last
27 02-Oct-2007 13:46 3 KB HuiLi to previous | to last
26 28-Sep-2007 19:29 2 KB KenDere to previous | to last Comment by KenDere
25 21-Sep-2007 16:49 2 KB JianSun to previous | to last
24 21-Sep-2007 16:40 2 KB JianSun to previous | to last
23 19-Sep-2007 06:42 2 KB David R Williams to previous | to last Comment by David R Williams
22 18-Sep-2007 14:46 1 KB JianSun to previous | to last
21 18-Sep-2007 14:45 1 KB JianSun to previous | to last

Page References

Incoming links Outgoing links
BigEISFitsFiles
BigEISFitsFiles

Version management

Difference between version and

At line 1 removed 2 lines
[{ALLOW edit EISMainUsers}]
[{ALLOW view Anonymous}]
At line 8 removed 3 lines
21/eis_l0_20070821_000022.fits.gz 118M \\
21/eis_l0_20070821_133226.fits.gz 117M \\
At line 12 changed 14 lines
*[Click to show more fits files|]
##22/eis_l0_20070822_023009.fits.gz 116M \\
##23/eis_l0_20070823_191420.fits.gz 94M \\
##24/eis_l0_20070824_001531.fits.gz 92M \\
##24/eis_l0_20070824_105128.fits.gz 120M \\
##25/eis_l0_20070825_015152.fits.gz 99M \\
##25/eis_l0_20070825_050529.fits.gz 95M \\
##25/eis_l0_20070825_093750.fits.gz 89M \\
##26/eis_l0_20070826_005803.fits.gz 97M \\
##26/eis_l0_20070826_055213.fits.gz 92M \\
##27/eis_l0_20070827_045520.fits.gz 97M \\
##27/eis_l0_20070827_170835.fits.gz 81M \\
##27/eis_l0_20070827_210723.fits.gz 79M \\
##28/eis_l0_20070828_005937.fits.gz 72M \\
*More fits files
**21/eis_l0_20070821_000022.fits.gz 118M \\
**21/eis_l0_20070821_133226.fits.gz 117M \\
**22/eis_l0_20070822_023009.fits.gz 116M \\
**23/eis_l0_20070823_191420.fits.gz 94M \\
**24/eis_l0_20070824_001531.fits.gz 92M \\
**24/eis_l0_20070824_105128.fits.gz 120M \\
**25/eis_l0_20070825_015152.fits.gz 99M \\
**25/eis_l0_20070825_050529.fits.gz 95M \\
**25/eis_l0_20070825_093750.fits.gz 89M \\
**26/eis_l0_20070826_005803.fits.gz 97M \\
**26/eis_l0_20070826_055213.fits.gz 92M \\
**27/eis_l0_20070827_045520.fits.gz 97M \\
**27/eis_l0_20070827_170835.fits.gz 81M \\
**27/eis_l0_20070827_210723.fits.gz 79M \\
**28/eis_l0_20070828_005937.fits.gz 72M \\
At line 28 changed one line
These fits files are mostly generated by study eg. "HPW001_FULLCCD_v2": doing "full-ccd" scan (with often over 80 raster positions).
These fits files are mostly generated by study "": doing full-ccd scan (often over 80 raster positions). The typical error message is: "unable to allocate memory to make array". However, the machine that running "eis_prep" has fairly big enough memory (4GB).
At line 30 changed one line
The typical error message is: "__unable to allocate memory to make array__". However, the machine that running "eis_prep" has fairly big enough memory (%%(color:#009900;)4GB%%). One way to let "eis_prep" keep running is probably to do only [DC-removal|] and [Abs calibration|], not to do [CR|] and [HP|] removal, :-(
Is there any way to do 'eis_prep' with these big fits files?\\
At line 29 added one line
Or do we need to suggest that normally do not use full-ccd scan with too many raster postions?
At line 33 changed 53 lines
----
Questions:
#Is there any way to do 'eis_prep' with these big fits files? Or,\\
#Do we need to suggest that normally try to avoid using full-ccd scan with __too many__ raster postions?
Regards,\\
[JianSun] [{CurrentTimePlugin}]
----
Hi, Jian
This is starting to become a big problem, and I think you're asking the right questions.
However, I don't think we should stop taking large data because the current software doesn't allow us to analyse it. This just means we need to think about . We want to take the best data we can and worry about analysing it later. You never know when an instrument/satellite can fail.
This sounds like a major issue for the team meetings in October.
In the meantime, if anyone has comments, please start making them! This is something that really needs to be sorted out.
--[David R Williams|http://www.mssl.ucl.ac.uk/~drw/], 19-Sep-2007
----
Hi David,
I agree with you and Harry that we should get science data first and then work out the solution, if there is problem to analyse it. So probably we need some modifications on "eis_prep" to improve the memory performance for processing these big fits files.
[JianSun] [{CurrentTimePlugin}]
----
The Hierarchical Data Format (HDF) was made for large files. There are a lot of good tools in IDL for working with HDF files.
--[KenDere|http://null], 28-Sep-2007
----
Hi, Jian,
This problem arises from the limited memory size that IDL can allocate. IDL running on a 32-bit computer can theoretically allocate memory of about 2.3GB (from IDL help files). However, according to my experience, it can only allocate memory of a little more than 1GB, which is not enough to process such a large EIS file with EIS_PREP.
Nevertheless, this problem can be easily solved on a 64-bit computer which running the corresponding version of IDL (also 64-bit). Of course, the computer should have large enough physical memory (maybe 4 GB or more). In this case, IDL can allocate large enough memory to run EIS_PREP to process the EIS data of large size. I have tested it using the file eis_l0_20070824_001531.fits.gz. I processed it with EIS_PREP successfully.
Another way, I guess, to solve this problem is to optimize the IDL code EIS_PREP and related ones, in order to reduce the memory requirement. This may be done by freeing the allocated memory in the code immediately after using it and reducing temporary variables if any. I am thinking about this because I met the same problem when I processed several EIS files with smaller size one by one successively. Therefore, I guess that when we call the code, the allocated memory in the run of the code is not freed after the call.
--[Hui Li|http://solar.pmo.ac.cn/lihui/] 02-Oct-2007
----
Regards,
JianSun
This page (revision-31) was last changed on 07-Dec-2016 14:14 by HuiLiTop