Update

Pushed a new implementation os starnet in TF2.x. The whole implementation is in one file starnet_v1_TF2.py.

I also created a few Jupyter notebooks for ease of use:

1. starnet_v1_TF2_transform.ipynb - loads and transforms an image.
2. starnet_v1_TF2.ipynb - more detailed example that loads a model and shows how to train it (really simple as well I think).

Weights for the new model can be found here.

StarNet is a neural network that can remove stars from images in one simple step leaving only background.

More technically it is a convolutional residual net with encoder-decoder architecture and with L1, Adversarial and Perceptual losses.

Small example:

Intro

Star removal using classical methods is a very tricky and painful multi-step procedure, which is hard to master and hard to get nice results from, especially in case of images busy with stars.

This neural net will remove most of stars from input image in one step, leaving only really huge ones, and leaving (well, hopefully) intact all other small bright things whose shape is significantly different from that of a typical star, like small spiral galaxies, fine details in nebulosity, HH objects, etc.

It is intended to be used by astrophotographers. Primary use is for background nebulosity enhancement in rich star fields, but it can also help in creation of nice starless image.

Literature

This code is partially based on pix2pix code and ideas from pix2pix paper.

pix2pix code: https://github.com/phillipi/pix2pix

pix2pix paper: Image-to-Image Translation with Conditional Adversarial Networks

Udea of using Perceptual Adversarial losses is from this paper as well as some other ideas:

Perceptual Adversarial Networks for Image-to-Image Transformation

Other papers I took ideas from or found useful during development:

Image De-raining Using a Conditional Generative Adversarial Network

Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections

Removing rain from single images via a deep detail network

Usage

Its primary purpose is to partially replace initial steps of star removal in tutorials, like one by Gerald Wechselberger, aiming to enhance nebulosity without pushing stars up. The tutorial itself was available under this link, but not any more, for some reason. Haven't found any newer links to it. Anyway, you got the idea.

Suggested Work Flow

The transformation by this neural net can be part of PixInsight/Photoshop processing work flow. Something like this:

1. Start from stretched LRGB image. Save as 8 bits/channel tif file.
2. Feed to StarNet.
3. Open output in Photoshop, correct some of the worst artifacts that will most likely appear in the image. If there are huge stars in the image left, you will have to take care of them in some other way.
4. Perhaps use some noise reduction (we don't want to push noise up).
5. Use resulting image to enhance nebulosity using some method (Screen-Mask-Invert for example) or enjoy the result.
6. ?
7. Profit!

Weights for the network

This repository contains only a code base needed to run the net, but does not contain all the weights (which are uploaded into LFS. Pre-trained weights are also available for now through my dropbox account because they weight too much (lol) - about 700 Mb. You need to download them and unpack into root folder of starnet (the one with all python scripts, not into some sub-folder) to begin using StarNet:

This will rewrite some files in this repo!

The weights are also uploaded into LFS, but depending how you clone the repo they might or might not download.

Some tips and tricks

1. Do not use heavily processed images for input. If star shapes are unusual (if star reduction techniques were used, too much sharpening, deconvolution, image was heavily resized, etc) the performance might be much worse than it could be. I am almost sure. Or maybe it will be better in your case, but that's unlikely.

2. This newral net was trained using data from refractor telescope (FSQ106 + QSI 683 wsg-8), so will work best for data from similar imaging systems. That's a bit of a bad news for many users of reflector telescopes. If you have long spikes in your images, then the net will not take care of these spikes very well, I tried and didn't like results too much. What you can do, however, is you can train the net a bit more using your own data. Just prepare one or two starless images from your data and run training for 20 epochs or so. This should significantly improve results and make the net much better for your data.

3. The point above is valid for all images, for which you are not getting good results. You can prepare a starless version even of a small part of that image (but not smaller than 256x256 pixels) and run training for 20 epochs or so on this image. This should improve quality of transformation of the whole image. This will take a lot of time, of course, but in the end you not only getting a starless image, but also train a net so it will perform better next time on a similar image.

4. Also it might help, for example, to make image brighter (darker) if it is unusually dark (bright), or things like that.

5. Sometimes the net will leave small stars in the output, if you feed it very busy image. In this case it is helpful to feed output to the net again.

Training Dataset

This is one part I'd like to keep for myself for now, but you can create your own dataset creating starless versions of your images. One extremely important note: the only difference between two images (original and starless) should be stars, which are replaced by background in the starless version. The rest of the image should be perfectly intact. If you will throw in a starless image which is super nice looking, but in a process of creation of this image you altered much more than just stars, this will only degrade network performance. Also be aware that quality of star removal by the net will never be better than that in your training set, so if you want a top-notch quality starless images, be ready to provide training images of even higher quality.

I left one training image to show organization of folders my code expects. Inside a folder named 'train' there are two sub-folders named 'original' and starless', both should contain equal number of images with identical name pairs.

Some technical info

Throughout the code all input and output images I use are 8 bits per channel tif images. This code should read some other image formats (like jpeg, 16bit tiff, etc), but I did not check all of them.

Prerequisites

Python and Tensorflow, preferably Tensorflow-GPU if you have an NVidia GPU. In this case you will also need CUDA and CuDNN libraries.

I tested it in Python 3.6.3 (Anaconda) + TensorFlow-GPU 1.4.0

Environment: I used Win 10 + Cygwin

GPU was NVidia GeForce 840M 2Gb, compute capability 5.0, CUDA version 9.1

  Modes of use:
  
  python.exe -u starnet.py transform <input_image> - The most probable use. This command will transform 
                                                     input image (namely will remove stars) and will 
                                                     create a mask showing changes regions. Output images
                                                     names will be <input_image>_starless.tif and
                                                     <input_image>_mask.tif
  
  python.exe -u starnet.py train                   - Use if you want to train the model some more using
                                                     your training data. This will also output logs and
                                                     showcases of training transformations in 
                                                     './logs' sub-folder.
                                                     
  python.exe -u starnet.py plot                    - Will plot graphs from log files inside './logs' 
                                                     sub-folder.
  
  python.exe -u starnet.py train new               - Use only if you want to train a completely new model,
                                                     erasing all older weights and other output, such as
                                                     logs. Use only if you know what you are doing!

  python.exe -u starnet.py test <input_image>      - This will create some test transformations of patches
                                                     of the input. Similar to transform, but instead of
                                                     transforming an entire image, will create showcases
                                                     of transformations. Fast option to take a look at 
                                                     possible result.
                                                     By default output will be in './test' sub-folder.

Couple more examples

More examples can be found here.

Original:

Starless:

FAQ

What is all this 'python-sudo-mumbo-jumbo'?

This whole thing works as command line program, which means that there is no graphical interface: you have to run it in a console using some text commands (like ones you see above) and it outputs text (and writes image files of course!).

The way you get on board with all this will depend on your OS and is hard to squeeze instructions into few words but very briefly:

1. For MacOS/Linux it is pretty simple. You should already have console up with python installed. The only thing you will need is to use pip to install tensorflow and probably few other packages via:

    pip install tensorflow
    pip install numpy
    pip install <whatever is missing>
   

2. For Windows it is a bit trickier because console is unusable to say the least. You can use it, but I prefer Cygwin. Next, you need to install python. I think installing Anaconda for that is by far the best option. After you got up the console of your choice and installed anaconda you use its native pip to install tensorflow (see above) and you should be ready to go.

The whole installation should not need more than installing few software packages and typing few command lines!

2. Where exactly do I put weights of the network?

All the files you download should be in one folder: all the files with extension .py (starnet.py, train.py, transform.py, etc.) should be in the same folder with weights for the network (model.ckpt.data-00000-of-00001, model.ckpt.index, model.ckpt.meta, etc.)

Some Troubleshooting

1. Error: 'No package named tensorflow'. Should be pretty self-explanatory: your python can not find tensorflow. That means you did not run pip to install it (pip install tensorflow) or something went wrong during this step if you did.

2. Error: 'ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory.' You are trying to use GPU version of tensorflow and you don't have CUDA properly installed.

3. Error: 'ValueError: The passed save_path is not a valid checkpoint: ./model.ckpt.' You did not copy network weights into proper location. See above.

Let me know if you have any other issues with the code. (Preferably through Astrobin)

Licenses

Code is available under MIT License, please review LICENSE.md file inside repo. Its very permissive, but no liability or warranty of any kind.

Weights are available under Attribution-NonCommercial-ShareAlike 4.0 International Creative Commons license.

In short: You are free to use and redistribute them in any medium or format, but only under the same license terms. You can transform, and build your projects upon them. You can NOT use them for commercial purposes. You must give appropriate credit for usage of these weights.

The weights are distributed on an "AS IS" BASIS WITHOUT WARRANTIES OF ANY KIND, EXPRESS OR IMPLIED.