DECam image quality

Something I am trying to understand is the darkening around bright objects for DECam data. Originally I thought these dark halos were the result of some kind of image processing, but I think I understand now that they are present all the way down to the raw data. I’ve tried to find information about this with google searches, but can’t seem to find any info. I guess I can’t get rid of them easily, but I am still curious just to know why it’s like that.

Hi,

I think you’re right that this is due to image processing – specifically, we try to estimate a spatially-varying background term, but around bright galaxies we overestimate the background, and oversubtract it, leading to ‘divots’ around bright galaxies. Our images go through two stages of processing – the raw ones off the telescope are available from the NOAO Science Archive (archive.noao.edu); these have “ori” in the filenames. Then they go through the “Community Pipeline” processing, where the images are called “ooi” or “oki”; oki has this flexible background model. Then we have the “legacypipe” processing – the DECaLS-specific part – where we re-estimate the background levels.

This is definitely something we’re hoping to work on for the next data release, but that won’t be for a while…

cheers,
–dustin

Oh perfect, ooi are what I need. I was about to give up after opening so many different ones and finding the divots (must remember this word) around each. ori are a little too raw, the others a little to processed… but ooi are juuuust right. Thanks as always, Dustin, dustin, or dstn, whichever you prefer.

Hey again, I would like to do more amateur image processing using these data, but it is driving me absolutely bonkers to have to search for the right ccd in each FITS file… I just have to ask if there is any easier way to retrieve individual ooi files from just CCDs that overlap a certain coordinate without getting all the unnecessary data?

Not expecting any good news, but I have to try…

Hi,

I thought that if you turn on the CCDs overlay, and click on the map, then you get a list of the CCDs overlapping,


and if you click on one of those "CCD: " links
(with URLs like
legacysurvey.org/viewer/ccd/decals-dr5/decam-425836-S12-g/ )
then you get a page that takes an extremely long time to load, but eventually shows you a thumbnail of the image, with links for the “image”, “weight”, and “data quality” images. These should be just single-HDU FITS images with just the CCD overlapping that part of the sky.

Please let me know if that doesn’t work or isn’t what you’re looking for.

cheers,
–dustin

Yes, I was doing that before I learned that those have areas around the galaxies with the background slightly oversubtracted. It makes it difficult for me to see what is and is not part of the faint outer halo of a galaxy. What I wish I could do is download the image just like that except without the divot.

I suspect there is no way, but thank you for always being so amazingly helpful, Dustin. I’m a total nobody and I always feel like I don’t deserve it.

Hi,

The “CCD” links give you the “ooi” files, which I think should not have such bad divots.

To recap, the images go through processing stages:

  • raw data have “ori” as the file type; these are what are read off the CCDs
  • these pass through the “CP”, the NOAO Community Pipeline, where they get flat-fielded, bias-subtracted, pupil corrected, and have a sky or background level estimated and subtracted.
  • the results are the “ooi” file types
  • these then go into the “legacypipe” code, where we do another round of sky subtraction – and I think this is the one that produces the bad divots – and coadd the images to produce the images that appear in the viewer
  • the coadds are named like “legacypipe-3340p002-image-g.fits.fz”

So, those divots are coming from the sky subtraction processing, and the confusing thing is that there are two different sky subtraction steps. My belief is that the second one (legacypipe) is doing the worse damage, so the “ooi” files should be better. That belief could be wrong! If the ooi files still look bad, then one would have to go back to the raw images, and I suspect re-doing that work would be pretty painful.

For what it’s worth, the raw images aren’t directly available through the web viewer – they’re available at http://archive.noao.edu/search/query , where you can also get the “ooi” files.

One other thing: these divots are of concern to folks working on bright galaxies, so it is something we’re trying to improve. In our upcoming DR6 release, we tweaked the sky subtraction method to be less flexible, with the hope that it will not produce such bad divots.

As for not deserving help: I disagree!! I’ve seen some of your work and it’s amazing! Also, this is a taxpayer-funded project with an explicit goal of providing data products that are useful to the astronomical community, and I would be very pleased if that included amateur as well as professional folks. (And I always try to keep in mind that the word “amateur” comes from the Latin word for “lover” - someone who does the work for love of the subject.)

Best wishes,
–dustin

1 Like

Unfortunately, the “CCD” links give the “oki” files. I am currently only able to go to archive.noao.edu after I find the oki filename from the legacysurvey.org page eg:
http://legacysurvey.org/viewer/ccd/decals-dr5/decam-535521-S3-g/

At the top it says c4d_160409_025257_oki_g_v1.fits.fz
I can change oki to ooi and then get the ooi file from noao. It’s >300 MB download, though, and then I have to fish out the correct CCD… the extra added steps are very discouraging.

Just to be sure, here is the file I get from legacysurvey, from the “image” link:

And here is the one I can get from noao:
[see next post, I am disallowed from posting two images in one post]

I used the bright one so it is easier to see, but it is still a problem for me with dimmer galaxies.

Thank you for the info about the Latin origin for amateur. I never thought to look it up. The word has lost its meaning and is largely used pejoratively these days, though the astronomy community is very good about avoiding pejorative use.

Judy

Hi Judy,

Wow, that was quick follow-up!

Okay, yes, I forgot to mention the “oki” files. There are different versions of the CP, and later versions produce both the “ooi” and “oki” files, where “oki” has a flexible sky model. From the images you have shown here, it looks like the “oki” files do indeed have divots around bright galaxies, while the “ooi” files look okay – am I reading your results correctly?

I believe we actually do have the “oki” and “ooi” files, so I think I could provide links to cutouts for both on that data download page. Would that do the trick for you?

Thanks for your perseverance in investigating and reporting this!

cheers,
–dustin

Yes, Dustin, I think you understand correctly, and I think I understand you correctly now. I would be very grateful for the link to the ooi files, if it is indeed the case that it can be added. It would relieve what feels like a heavy burden, though perhaps my brain is exaggerating that a bit.

Judy

Hi Judy,

I originally added these links to download just a single CCD rather than the whole DECam images because it is such a pain to download the huge files and then figure out which extension is the right one, so I feel your pain!!

I just added a couple of things that I hope will help:

It’s easy enough to add more links (eg, to weight maps or data quality masks) if that’s helpful; please let me know.

cheers,
–dustin

1 Like

You’re the best, Dustin. This is perfect.

I never figured out how to make use of weight maps, but sometimes the data quality masks are useful. I wonder if people other than me would find the links useful. I don’t think I am the best person to decide whether to add those links or not. They certainly wouldn’t bother me if they were there.

Great – I just added links for both.

cheers,
–dstn