The ALASKA competition is now over. We would like to thank all of the users that made this adventure such a great experience. While we are finishing the paper that will summarize the major results of the competition, we would like to share some of the salient findings of our analysis of the submissions.
Most competitors failed to get across the Teklanika River and sunk. The only team to survive this long journey is ABBA, from Binghamton University. Armed with a novel and powerful attack, they were able to face the hardships inherent to a life in the wild and get to the top of the podium.
Unsurprisingly, deep learning has been the weapon of choice for most participants and was used by every member of the Hall of Fame.
However, despite some impressive results overall, our analysis of the performance of classifiers on the different types of diversity offered by the competition indicates that much of the work towards robust steganalysis remains to be done.
Finally, without any further ado, let us reveal to you the "ALASKA survivors Hall of Fame" :
Users / teams
PMD5 (website)
PMD5 (real)
Pemin
FP50
Binghamton University
24.37
25.20
14.48
0.71
Shenzhen University
50
51.60
25.20
5.86
3188960009
54.93
53.8
26.33
7.67
375790798 / veyron_gz
53.35
54.2
25.78
7.67
Those were the only competitors to cross the bound of PMD005 at 55% ; we have kept those competitors because, interestingly, the team ranked third was actually ranked 5th on the website with scores computed over subset of images.
The ALASKA steganalysis challenge is now over ; we wish to thank you all for your participation.
We look forward to seeing you in July in Paris and ... perhaps for the next steganalysis challenge.
(written on: March, 29th, 2019)
Will Binghamton Team win so easily ?
Right at their entrance into the game, Binghamton team (yyousfi1) slammed a 43.8% on Nov. 10th which they turn to 37.5% 10 days later. Since then, no other team went below 52 %, leaving an enormous margin of almost 15% between their top followers and a "first shot".
The question we will soon answer now is the following: are some teams "bluffing" ; it is indeed completely possible to play "offline", to work to improve the performance of a steganalysis method while not submitting any answer. The goal of such strategy would be to look at what is going on and either to create an "effet de surprise" or to play only if it is worth it.
With two weeks remaining, we will soon have a pretty clear answer to this question.
Let's us just wait and see what will happen as we approach the home stretch. A final word to remind the competitor with the best scores that they can submit their paper to the ACM & IH&MMSec conference (see the call for paper here or the pdf version there ). The submissions are expected before March 15th and the submission will be evaluated on both the performance, the methodology and the originality to overcome the problem of steganalysis "into the wild".
(written on: March, 1st, 2019)
ALASKA competition rages again
It has been a while (almost 2 and a half month) since the last review on ALASKA challenge.
Why such a long time ? Because after Binghamton team, USA, slams a record on November 10th, beating up previous submissions by more than 10% missed detection rate (for 5% false alarm rate).
Indeed, "yyousfi1" increased it score by almost 6.5% by the end of November and then ... not many remarkable events happen. The followers hardly improved their scores.
Then by mid-January we have had incremental improvements by several users, especially "2016130231" who now is ranked third (right behind Binghamton teams) ahead by a non-negligible 2%.
The competition starts raging again to take the third place on the podium with the user "375790798" (currently ranked fourth) regularly improving its submissions ; By the mean time, a newcomer, user "3188960009" reached the fifth place with only 4 submissions and is now only 2.5% away from the podium. Another competitor "veyron_gz", who is striving to reach the top, is still in the game with 55% missed detection rate.
This show us that the "following" competitors did not get sick seeing the results from the dominating teams from Binghamton but instead get revolved recently ; will they manage to challenge the leaders ? Hard to tell from now. For sure the competition increases and it is very exciting to see who will be on the podium in a month and a half at the deadline ... clock is still ticking.
Again, let us have a look at a glance on how much submissions are close to each other.
Surprisingly, submissions from the dominant teams from Binghamton are not so very similar. On the one hand, the submissions from "yyousfi1" from November and January are very similar. One can note over 95% of the same image in the first and last 500 indices (most like stego and cover images resp.). On the other hand, the submission from "jbutora1", also from Binghamton, who reached 39% missed detection show more than 25% of different images labelled as the 500 most like stego (first 500 indices)
Behind, when looking at the four "followers" two groups can be identified. First, the user "2016130231" (currently third, from Shenzhen University, China) and the newcomer user "3188960009" (currently fifth) are very similar also sharing more than 95% of the same images ranked as both 500 most likely covers and stego. Those submissions, however, do show less than similarly 75% of the most likely 500 covers and stego with users "375790798" (currently ranked fourth) and "veyron_gz" (currently sixth).
Very surprisingly, those users "375790798" and "veyron_gz" reached their best score with exactly the same submission ... Looking at their IP addresses most of their connection are from the same locations ...
This is an opportunity to remind that the ALASKA challenge rules forbid using different account to submit more than one answer per hour and that breaking this rule may results in being banned from the IH&MMSec special session ....
Last, let look at the activity of the challenge : we now have (by January 30th):
The number of users is now 194 ; as compared to the 183 we had on November 15th, it seems that most of the interested researchers are already in ;
ALASKA challenge reaches 320 submissions (about +50 per months) from 41 users (+9 since the last blog update in mid-November) ;
We measured 5500+ unique visitors (slightly decreasing by 25% to about 26 visitors per day).
Stay tuned for updates about what comes out from the last efforts of the striving competitors.
(written on: January, 30th, 2019)
Users get seriously into the ALASKA challenge
The ALASKA challenge has been started two months and a half ago and it has been one month since the last review.
Over this last month, there has been an important increase in the activity and an impressive improvement in best submission's results.
Indeed the "UPPER BOUND" has been broken on the 24th of October by user 2016130205 with about 64% missed detection (for 5% false alarm rate).
Then, the 60% missed detection barrier has been reached very quickly by user 1185205583 before making an impression jump by improving all previous scores by 5% reaching a missed rate of 55% on the 3rd of November.
This remained the best submission until the shattering entrance of the user yyousfi1 who slammed a submission with as low as 44% missed-detection rate.
The current ranking shows, among other, three groups of very tight competitors (scores given in % of missed detection for 5% false alarm rate):
yyousfi1, from Binghamton University, USA, is leading by far with 43.81% ;
Three competitors, all from Shenzhen University, China, are struggling behind: 1185205583 hits a 55.34% tight with 2016130205 reaching 55.34% and slightly above 2016130231 with 56.59%
Eventually a group of four other competitors (from unknown institutions) are about 6% behind and very close to each other: veyron_gz with 61.56% , 763244161 with 62.19% (and only 3 submissions!) , hoopabc with 63.28% (only one single submission) and jishenzeng with 63.41% (who did not submit since 1st of November)
Interestingly, we measured the similarity of those submissions by counting how many images they have in common among the 500 firsts and 500 lasts (hence, that are labeled as the 500 most likely stego and 500 more likely covers).
It is striking that yyousfi1 share only about 35% of images on this subset with its three followers.
On the opposite, the group from Shenzhen University has more than 80% images in common in those subsets. The group of four followers competitors is more heterogeneous with a remarkable submission from jishenzeng that share less than 20% of 500 most like covers and stego with the current leader (while reaching a pretty decent score).
Eventually, let us finish with a closing remark on the activity on the ALASKA challenge, as of date we have:
183 registered users (+42 in one month) ;
176 submissions (+105 in one month) from 32 users (50% increase as compared to the past month);
3700+ unique visitors (hence, roughly the same average traffic of about 35 unique visitors per day);
(written on: November, 17th, 2018)
Results after (a bit more than) one month
It has been a month and a week since the ALASKA challenge started and it is time for a first review of competitors' results.
After a sort of warm-up phase during which the results were below what we stated as a "lower bound", recent submission reached impressive results, getting closer and closer to the "upper bound".
One can also note that Chinese competitors are leading the challenge by far with three tight submissions around 30% of missed detection for 5% false alarm rate.
Interestingly, looking at those three best submissions the current leader (jishenzeng) and the current third (1185205583) provided very similar results ; their first 500 images, ranked as the most likely stego, overlap by over 90%. Looking at the current second competitor (2016130205) this overlap is below 50%.
The challenge also seems to be quite a success up to now with ; as of the writing of the blog entry, we have:
141 registered users ;
71 submissions (hence almost one per day) from 20 different users;
2576 unique visitors (just above 1000 per month) from 15 different countries;
We are truly looking forward to seeing how long the "upper bound" will hold and if those recent breakthrough submissions stimulate the competition.
Eventually, note that we are currently preparing a "technical" paper to present how we assess the difficulty of the challenge and set those "upper" and "lower" bounds.
(written on: October, 14th, 2018)
Kickoff of ALASKA contest is postponed
Unfortunately, we have to postpone by a few days the beginning of the ALASKA contest.
The two main reasons that lead us to postpone the beginning of the challenge are the following:
First of all, we have noted a minor bug in the conversion script. Since we wanted to provide datasets that will not be changed over all the duration of the challenge and that have been obtained exactly in the same way as the testing set we have to generate once again all the training jpeg data ;
The second issue is that, due to this minor bug, we are not 100% confident about the evaluation of the difficulty of the challenge we have made so far. We, therefore, would rather give us a few more days to ensure that the contest will have the right difficulty, neither "piece of cake" easy nor infeasible.
However, we have some good news on the "plus sign". First to take into account the kickoff delay the challenge is extended by 15 days. We also have (eventually) uploaded the complete and final version of the RAW dataset (made of 50,000 images).
Second, we are currently preparing the jpeg version of the training set and we will provide, as per users request, stego images. So the training will be made of 50,000 cover and 50,000 stego images.
Eventually, as per other users request, we will provide uncompressed version of the jpeg images for those we might be interested in using the "wild dataset" for spatial domain steganalysis, forensics or any other research topics.
We sincerely apologize for the inconvenience this delay may cause and thank you in advance for your understanding.
(written on: September, 1st, 2018)
A Midsummer Update
ALASKA has been taking some vacations over the past days, but it is now back for setting everything for the beginning of the competition by September the 1st.
The server has been restarted this morning for a quite important update.
An important update ? What's new for now ? check it out:
Perhaps the most important changes, we have slighty modified the server configuration such that all web-browser, regardless their "behavior" with respect to certificate, are able to access the data proposed for download ;
The LeaderBoard has been changed to hide the full email address of users and to add the submission time (We do thank Andreas Westfeld for this clever suggestion) ;
The material for downloading RAW images has been updated; we have especially corrected a minor "bug" in the conversion script that may lead to full output full resolution images.
Similarly, the proposed JPEG training sets have been updated since our dataset was corruped with this bug and, hence, contained full resolution jpeg images. If you already download the jpeg training set you can either redownload the updated clean version or simply ignore the less than 1% images with high resolution.
Note that another important update will be made by the end of the week because we are currently updating the RAW image dataset. We increase the dataset from 41,000 images to approximatively 47,000 almost reaching the targeted size of 50,000 highly diverse raw images.
We will keep you update with any other changes.
(written on: August, 19th, 2018)
First blog entry
With, a day or two, of delay on the initial schedule, we are proud to official announce that the ALASKA steganalysis contest is now open. We will use this page to keep you up with, for instance, information on the number of submission and the countries of the competitors. During the warm up phase (all along month of August 2018) we will also keep you up with advances on the creation of the dataset and possible patches that we have to make.
For any remark, question or help, feel free to contact us at: alaska@utt.fr.
We truly hope that you will have as much fun as possible during this ALASKA contest and that innovative steganalysis method for highly heterogeneous source of images will comes out. See you at IH&MMSec'19 in June in Paris.
