DSS2 all sky survey on SKY-MAP.ORG

[keeperlink]

Finally, after several months of sleepless nights I managed to create semi-automated process to color balance and match DSS plates into more-less solid mosaic (I'm working on DSS2 all sky survey for the SKY-MAP.ORG project)

I processed all sky at 1/16 of full resolution and half way processing at full resolution.

Meanwhile I have a question for astrophoto gurus:

I'm pretty much happy with most of the areas, nebulae, i.e. Lace-work nebula - http://server3.sky-map.org/?img_bord...mg_source=DSS2
Flame nebula - http://server3.sky-map.org/?img_bord...mg_source=DSS2
dark nebula above Antares - http://server3.sky-map.org/?img_bord...mg_source=DSS2

But most of galaxies look too bluish
Andromeda is more-less ok to me - http://server3.sky-map.org/?img_bord...mg_source=DSS2
But other galaxies...
i.e. http://server3.sky-map.org/?img_bord...mg_source=DSS2
http://server3.sky-map.org/?img_bord...mg_source=DSS2

The question is how too adjust my process to fix galaxies, but leave good areas unchanged or changed insignificantly. It could be some simple change in color calibration function. I can spend another month to figure it out, but for you it might be something obvious.

There are still few significant edge plate problems in the mosaic, but I have some solutions to solve it. But if you zoom out you'll notice quite spotty background. I'm still looking for a way to solve it. Any ideas?

Thanks,
Sergei

[RickJ]

Looking at the Histograms for the M101 image I see a lot of problems. Obviously the galaxy was processed using two different DSS plate combinations for the two halves of the galaxy. This is especially obvious when you look at the red channel alone in my second image. Also in that image look how different the three colors are. The red image is very poor though the DSS red plate image looks fine to me when I downloaded it. Something is going very wrong in your processing.

I'm going to guess it automatically sets the background level and the uses some arbitrary curve to stretch the rest of the image. M101 is on one DSS plate so your cutting it in half has to be due to some automatic process you are using. It obviously processed one half very different from the other. Even so it is pushing blue far more than red as shown by the three black and white images of each color channel The gamma applied to the red image is way too severe as well. Neither are treated very well however. The process obviously killed the red channel badly on the right side and treated the left side far worse. You need to find what's happening to that red channel, its on vacation. Then tone down the push to the blue as its too much.

In the three channel view all three images should look much the same. See my image in the third attached image. The three images all look virtually identical. That's what you need to strive for in your processing of the DSS plates. Get the red and blue the same and then make the green. It will take a good setting of the background cutoff and separate curves for the red and blue plates. Work only from the FITS images. The GIF versions can vary widely in quality and have very limited dynamic range.

I see similar problems with the nebula images except it is reversed. There the images have lots of red but no blue to speak of. Again the three histograms and the three RGB images in black and white should look much the same. They don't come close to this goal.

Not knowing the routines you are using I have no way to know what's causing this. That you'll have to figure out.

Rick

[keeperlink]

Hi Rick,
Thank you for your reply.
Yes, I have automated process to calibrate DSS plates. There are around 4000 plates (500 and 200 MPixels each) and its significantly vary in intensity. So the process basically determines two major parameters per plate - background level and top histogram cut out level.
For background the process chooses max histogram (it is also quite challenging task as plates randomly vary in intensity from side to side, so I had to produce some compensation masks for each plate);
For top level cut off the process takes the histogram and cut at at some constant percent like 99.5%. That's where I believe the major problem is. Looks like the astro images give quite stochastic results here. Is there better solution to decide on top cut out parameter for astro images?
You image looks beautiful!
Did you use additional histogram correction (along with min max)?

Thanks,
Sergei

[RickJ]

I'm not a programmer so don't know how to go about what you are trying with any automatic results. Yes, their plates have gradients, sometimes bad ones. I know of no way to automate their removal except by a crude pseudo flat -- see below.

There are gradient plug ins for photoshop but they take a lot of skill by the user to use. Nothing close to being automatic. Gradients can run several directions. It would take a strong AI ability to detect those and handle them correctly. Once those are removed then it's pretty easy to set the background level. But since their exposures of red and blue aren't consistent AI would also be needed to adjust the histograms in red and blue until they are equal then make the green plane after this. Then where processed images meet further work will be needed to prevent the problem seen in M101 whose red halves were processed at very different cut offs. Both way too high, especially the one on the left.

You are trying to do something that's never been done so you'll be the expert when this is over.

Those making individual images from the DSS plates don't do it with any automation that I'm aware of. Most do it as the article in the January 2008 Astronomy article details. All manually. Its very subjective. Trying to get this to match across a wide area would be difficult. I find it difficult to match two fields I've taken the same night and exposed exactly the same.

Seems to me I've seen mention of some software that does match moon shots from web cams for making them into seamless mosaics. Not doing this I've never paid much attention to those threads. I have no idea if that might help match your seams or not. These are far larger images than a bunch of 640x480 web cam images. You are many magnitudes beyond that!

One step that might help with your gradient removal problem is to use pseudo flats. This is not nearly as good as using a gradient plug in but could help automate the process with poorer results. Normally this too needs manual inputs but they tend to run around the same values so you might try those.

Make a duplicate layer, apply a median filter. The size of this depends on your image scale. If it is about 1" per pixel I'd use about 80 pixels as my first guess. This blurs out the stars. Galaxies and bright parts of nebula remain. I then apply a Gaussian blur of about the half the pixel size, say 40. Then I select the bright areas that remain from the main objects and fade them with a large feather. This leaves mostly the gradients in the image. Combine the layers using the difference option and set it at about 75%. I find most will pretty much vanish between 70 and 80% but do set it by watching the results. You don't have that luxury so maybe 75% will sort of work.

A gradient plug in works far better but I don't see that being automated but an action for the pseudo flat process could be recorded in PhotoShop to do it automatically. Would still leave some but is better than just applying a cut off to the whole image as you are now doing.

How you match all the seams after this I have no idea. Maybe someone else does.

For matching histograms so they are equal some normalization programs claim to do this. I've preferred to do it manually. CCDStack is one that comes to mind. How well it can be automated however I don't know.

Rick

[keeperlink]

Thanks, Rick

I'm on the way of significant algorithm improvements to solve the color disbalance problem as well as working on better de-edging solution.

Another thing I'm doing now is improving scaling down method. Initially I was using simple nearest-neighbor method to scale down the images. I tried more advanced techniques and got dramatic visual improvements.
I was playing with bi-linear, bi-cubic and average interpolations. I have several layers of pre-processed images, every next layer is scaled by factor of 2.
bi-cubic interpolation gave me more sharp images, but stars I think are look "too sharp". On other hand average method produced bit blurry output, but IMHO stars are look better. I just want to have your opinion - which one of those two methods looks better
(server3 is bi-cubc; server5 is average):

http://server3.sky-map.org/index2.js...+Nebula&zoom=7
http://server5.sky-map.org/index2.js...+Nebula&zoom=7

http://server3.sky-map.org/index2.js...ect=M45&zoom=7
http://server5.sky-map.org/index2.js...ect=M45&zoom=7

http://server3.sky-map.org/index2.js...ect=M81&zoom=6
http://server5.sky-map.org/index2.js...ect=M81&zoom=6

http://server3.sky-map.org/index2.js...ect=SMC&zoom=6
http://server5.sky-map.org/index2.js...ect=SMC&zoom=6

http://server3.sky-map.org/index2.js...Antares&zoom=6
http://server5.sky-map.org/index2.js...Antares&zoom=6

http://server3.sky-map.org/index2.js...ect=M31&zoom=6
http://server5.sky-map.org/index2.js...ect=M31&zoom=6

http://server3.sky-map.org/index2.js...+Nebula&zoom=6
http://server5.sky-map.org/index2.js...+Nebula&zoom=6

http://server3.sky-map.org/index2.js...=IC1396&zoom=6
http://server5.sky-map.org/index2.js...=IC1396&zoom=6

server3 - scaling using bicubic interpolation
server5 - scaling using average interpolation

(at zoom=5 and up (4,3,...) scaling data is using different sources and methods, so don't compare it. zoom=5 is shown as 30' on scaler in bottom left corner, zoom=4 ~ 1deg)

Thanks,
Sergei

[keeperlink]

We just released new DSS2 images on sky-map.org. There are many improvements in color balancing, plate edges fixes, background normalization.

The idea was to put DSS2 data to limit to catch every fade object. Here are some spots:
http://www.sky-map.org/?ra=18.66&de=...mg_source=DSS2
http://www.sky-map.org/?ra=20.93855&...mg_source=DSS2
http://www.sky-map.org/?ra=20.37386&...mg_source=DSS2

We still have some plates with bad color balancing and edges problems, but it much better now then previous version (old version is still available at http://server3.sky-map.org:8080/ ). We developed quite complex tool for fine-tuning dss plates individually, so we continue fixing the survey spot by spot.

It would be interesting to hear opinions and suggestions from experienced astro photographers.

Thank you,
Sergei