Greetings Community!
I am finding that this hobby is not only expensive for gear, but (exceptionally) expensive now for storage space. I can fill out a 4TB drive in one evening (I sometimes have 4 scopes going). What do you do with your stacked FITS files? How about the stacked (prior to stretching and editing) data? What about completed images? I do have a 32TB RAID 6 which I back up, but even this is getting somewhat full!
Thank you.
I keep my lights and their corresponding flats as well as the Master output folder from PixInsight. They are kept by year by target on my NAS. I went to some expense and a learning curve to get to this point. The problem is I have no idea why. I guess the concept is that you can add data and I do in fact do that often with more current data sets, but I probably have a few TBs on the NAS of lights and flats shot with older cameras and different telescopes that I can’t imagine ever stacking and processing again.
HANG ON…..I just had a thought. Much of that data was stacked and processed very early on in my journey, before I knew what I was doing and before I had the tools I do now. I wounder what’s in there that is worthy but never had a chance because of me? Time to revisit some old data. Thanks Kevin W.!
I keep all the lights and flats, everything else goes in the bin. It is a large amount of data but that’s the ballgame. I fill a 1T drive about once a year and that’s with a 585! I can’t imagine what it must be like for folks shooting full frame at a remote site.
If I had to down mode this further I would keep the raw stack(s) and dump everything else.
Like others, I keep lights and their associated flats sorted by target and night. I tend to only keep pre-processing files while I am actively working on a project, but once I am done, I will delete things like debayered and registered files and then only keep the raw lights, their flats, and whatever drizzle integrations and rejection_low maps are produced.
I also separately keep dark and bias calibration files at various gains and temps in an archive that I update once in a while. I am able to keep things relatively under control across 2 2TB nvme drives, and I also have a 14TB backup drive that contains backups of the active data along with archived data from prior years. If I end up revisiting a project to add data or just re-attempt with better techniques, I will transfer old archive data to my nvme drives, and will go through pre-processing again.
I’m keeping culled calibrated Lights, the master after stacking, and the photoshop file. I’m unclear why it would be advantageous to keep flats, but perhaps I’m doing it wrong.
I have a NAS and back it up online for about $250/year for 10TB.
I have digital photography from 20 years ago I’ve re-edited with improved software. I don’t feel that the best images are ever finished. Processing in Astro will likely improve substantially.
Tangentially related: The best AI are bad at the more complicated pieces of astro software, in my experience. But I’ve had a paid AI study an Adam Block video on GHS (pixinsight) and then walk me through the recommended process. AI Adam Block. Next I’m going to select the best WBPP training sources and learn what the heck all those options do.
Like most people I much prefer to learn by hands on with my own stuff. I would like to edit my photos with Adam Block looking over my shoulder. But AI Adam Block is pretty close. As I do this sort of learning more and more I’m actually doing patreon and similar more often. Probably because I find the original teaching material more valuable with an AI boost.
donld · Aug 28, 2026, 07:15 PM
I’m keeping culled calibrated Lights, the master after stacking, and the photoshop file. I’m unclear why it would be advantageous to keep flats, but perhaps I’m doing it wrong.
If you are going to keep your lights then you’ll need the flats of you ever want to re-stack.
Hello
I keep the lights, flats and darks- Also keep the pixinsight proyect in order to recall workflow if necesary.
I leave raw data on the capture computers during a year just in case.
I store everything in a local 10TB NAS server and back it up every month. If necesary to cleanup space I transfer old data to external disks and keep external disks in safe place then erase old data from NAS.
I keep lights, calibration subs, registered subs and masters after finishing projects. They go on a NAS with 6×20TB drives on RAID 5. 34TB used so far with 10 years of remote imaging. Usage has speeded up the last couple of years as I’ve gone to 120 second LRGB subs on one of the scopes and I have two remote sites now.
I’m keeping the old CCD subs mostly for nostalgia purposes and because it takes up so little space. In the old days, even with a medium format sensor, the subs were only 16 mpix and LRGB were 10 minute subs and NB was 20 or 30 minute. So you would have a project about 1/10 the size of today.
Kevin W. · Aug 28, 2026 at 02:27 PM
Greetings Community!
I am finding that this hobby is not only expensive for gear, but (exceptionally) expensive now for storage space. I can fill out a 4TB drive in one evening (I sometimes have 4 scopes going). What do you do with your stacked FITS files? How about the stacked (prior to stretching and editing) data? What about completed images? I do have a 32TB RAID 6 which I back up, but even this is getting somewhat full!
Thank you.
4 TB in one evening is an exceptional amount of data. Are you doing short exposures with an IMX411 sensor? I run an IMX455 sensor on two telescopes in Chile and I generate under 10GB/night using 3 minute exposures. So, question one is: How are you generating that much data each night?
I was going to tell you to get a large NAS system but it sounds like your “32TB RAID 6” might be the same thing. 32 TB isn’t huge but if you really are churning out 4TB/night, that system is only going to last for 8 nights! Getting a larger NAS is a good idea but at 4TB/night, you are going to need an exceptionally large system.
To answer your question more directly, here’s what I do. I keep all of my raw data on my NAS system and I never throw any of it out. I have a local PC connected to the NAS where I do local calibration and stacking. I save the masters only to the NAS system, which is from 3-7 files. I toss out the calibrated and registered files. I then download the masters to a local USB drive that I use for final processing. My processed files can chew up a lot of space and I don’t process a huge number of files each month so a 5TB USB drive will last at least a year or two. The weakness in my personal system is that I’m not very rigorous about backing up the processed files to my NAS system very often so I could lose a lot of stuff if I have a drive failure. My NAS system is totally backed up with a second, off site NAS so it’s safe so all I need to do is to be more careful about backing up my USB drive data.
John
I cull my lights, delete the rejected frames.
Stack and calibrate everything and stack them.
Then, i convert the fits lights to compressed xisf files, and zip them.
I keep the zip of the lights and the master dark/flat/light.
I have a similar approach as others.
I vet my raw lights via Subframe selector and place into a folder titled as SFS
The original raw lights folder is then deleted.
I keep a flats / dark flats folder by date / camera / gain / temp etc.
After running WBPS I delete all the folders except the “master”
I have a NAS that backs up my imaging computers nightly.
Annual data usage would be ~ 1 to 2 TB.
What I find critical is having a good repeatable and sustainable file naming convention.
I keep all my light and calibration frames but don’t keep the masters. I keep my raw data on my NAS, which has plenty of storage.
This might change in the far future, but for now this works for me.
If/when I want to go back and rework a project, I will re-stack the frames because chances are, the software has improved since my previous attempt, or my skill level has increased, and I will get a better result by re-stacking.
Kevin W. · Aug 28, 2026, 02:27 PM
Greetings Community!
I am finding that this hobby is not only expensive for gear, but (exceptionally) expensive now for storage space. I can fill out a 4TB drive in one evening (I sometimes have 4 scopes going). What do you do with your stacked FITS files? How about the stacked (prior to stretching and editing) data? What about completed images? I do have a 32TB RAID 6 which I back up, but even this is getting somewhat full!
Thank you.
I keep my lights, flats and the Master folder only of the WBPP integration from Pixinsight and delete all other stacking artefacts. Also retain some of the main artefacts during processing e.g. SPCC or Starless images incase I get to learn some new techniques or scripts to reprocess the images especially on Mono LRGB or SHO images.
But even then each image folder goes anywhere from 10-40 GB with all the data and the external HDD/SSD get full in no time
Am also deleting raw data of bad captures incase they have too much noise or some issues during imaging as these have no further use.
The challenge to clean up is not just with HDD /SSD but also ASI Air as mine has only 256GB capacity.
So far I have always saved originals, flats, master stacked files and the PixInsight project files.
Recently I have been wondering if keeping the PI project files is worth it. A little while ago, I re-opened one from two years ago or so, and so many processes/scripts don’t work anymore, work differently now, etc. I do write down for each image my processing workflow, so re-creating it with the latest tools, current skills etc is probably a much better approach.
I’m curious about practices in keeping telescope performance data and the potential value. I’m undecided on this topic. The recent logs are valuable, but do I want stats from six months ago? I’m undecided. The one area I think I will keep for sure is data related to temperature focus compensation.
That’s a serious amount of data you’re generating there. Seems like you’d need a business level solution if it’s feasible.
FWIW, I keep all culled lights, flats, final stacked image, PI project, final png, and final tif/tiff file on the mounted drives on my homebrewed RPi NAS. What the drive’s total storage TBs are peanuts compared to what you will need.
I keep the master stacked files and delete all the lights and flats. I find them useless after the image is stacked. I run 7 imaging rigs and have multiple RAIDS and NAS, but I still fill up drives fast.
The challenge with any storage solution is proper back-ups. Let’s assume the ‘golden’ 3-2-1 backup rule. Three copies, two different media, one off-site. For every file you keep, you need to have three times as much storage space available. A 20TB NAS will need 40TB of storage space somewhere else. And with storage costs essentially doubled over the past year, ‘nice to haves’ and ‘must haves’ may need some reconsideration.
I’ve been running into this issue after having started remote imaging almost a year ago.
Initially I was keeping nightly masters and calibrated files, and ditching the raw files (trying to map which flats are for which in the past would be beyond my organisational skills).
Now I ditch the nightly masters as well.
I was keeping the calibrated lights figuring I could add data over multiple seasons. I tried it recently and it didn’t work well - I’m not sure why. Interestingly though, when I used the final stacked masters from last season and added them to this season’s final stacked masters for a target as a “mosaic” (even though its the same target), it worked a lot better.
So now I’m chewing over even ditching historical calibrated files and just keeping seasonal masters which I can “mosaic” for deeper integrations.
I keep lights, master flats and darks plus original flats and darks, master bias, pixinsight project file, master tiff and a few intermediate tiffs such as a starless and stars, if I made them. All on a 24 tb external that is then backed up to a cloud storage service, which is helpful as it provides an off site backup.
This leads me to the question, what do you do when drive space, no matter how large is exceeded? Is there a physical media storage solution that’s dependable for the long term, 10-20 years or more? Most of us can’t keep adding servers and external drives have a finite life span.
Tony Gondola · Sep 1, 2026 at 02:53 PM
This leads me to the question, what do you do when drive space, no matter how large is exceeded? Is there a physical media storage solution that’s dependable for the long term, 10-20 years or more? Most of us can’t keep adding servers and external drives have a finite life span.
I don’t think anything could be depended on for that long; even if it could, there's still no guarantee you would be able to read the data with the hardware available. Digital storage is a game of backing up, backing up again, and transferring to new backups for the foreseeable future. I have slides from my film days that sit in file cabinets and are as archival as it gets, but when I compare the image quality from those days to now, the slides don’t hold up that well. When I run out of space, then serious culling of the herd will have to occur, and maybe it won’t matter because the recent data will be better? I don’t know.
I guess that’s the thing about living in a digital world that no one talks about. The degree of effort and resources involved to preserve history. Not just the big stuff but the little things. With tangible media, information could be easily passed down through the generations. I know great gaps have been ripped from by own life because I can no longer read a CD or no one makes that tape drive anymore. I’m so thankful for family history that’s been preserved through letters and photographs. Sadly, I think that’s going the way of the DoDo.
Maybe more of us should be turning to prints, at least of what we perceive as our best work at the time.
I save the final image as a TIF file and rid of everything else when I’m done processing the image. I’m rarely motivated to go back and reprocess old data, even if new tools come along (e.g. ML5). If I want to revisit a target, I’ll collect fresh data.
Kevin W. · Aug 28, 2026, 02:27 PM
I am finding that this hobby is not only expensive for gear, but (exceptionally) expensive now for storage space. I can fill out a 4TB drive in one evening (I sometimes have 4 scopes going).
Sorry for the comment, but you have up to 4 scopes running and you find storage “exceptionally” expensive??
That’s an interesting “scaling” at play here. 😉
Joke aside, the topic is well worth bringing.
Clear nights are few and far apart so I do not have to deal with insane amounts of data on a yearly basis, but at 50-60MB/image from a DSLR it still fills up a drive quick. I therefore remove calibration files once the unprocessed master is satisfying enough, and might throw out the lights too (I don’t think I ever added data to an old target). I retain the masters and saved steps from PI, but that’s it.