ACM regionals and finals questions..

Post here if you don't find any other place for your post. But please, stay on-topic: algorithms, programming or something related to this web site and its services.

Moderator: Board moderators

User avatar
abishek
Experienced poster
Posts: 131
Joined: Mon Dec 15, 2003 5:41 am

ACM regionals and finals questions..

Post by abishek » Tue Apr 12, 2005 8:45 pm

the acmicpc live archive contains the names of all the questions correctly. However it doesnot allow us to view them. Isn't this kind of ridiculous?
Also as the ACM has officially regonized the UVA site during this years finals, why don't they provide the test cases for all the problems in the regionals and the world finals?

Per
A great helper
Posts: 429
Joined: Fri Nov 29, 2002 11:27 pm
Location: Sweden

Re: ACM regionals and finals questions..

Post by Per » Tue Apr 12, 2005 10:27 pm

abishek wrote:the acmicpc live archive contains the names of all the questions correctly. However it doesnot allow us to view them. Isn't this kind of ridiculous?
Well, someone has to format the problems into html documents. Just getting the names of the problems isn't much work, but converting them into html is.
Also as the ACM has officially regonized the UVA site during this years finals, why don't they provide the test cases for all the problems in the regionals and the world finals?
The regionals run themselves, and its up to the individual regionals if they want to make the test data public or not.

For the world finals, it has always (I think?) been the policy of the ICPC not to release the test data in any form because they want to avoid the kind of controversy you get when, a month after the finals, someone discovers an error which would have completely changed the result of the contest.

shahriar_manzoor
System administrator & Problemsetter
Posts: 399
Joined: Sat Jan 12, 2002 2:00 am

hmm

Post by shahriar_manzoor » Thu Apr 14, 2005 3:44 am

I think ICPC may change its poilicy and release an unofficial data this year. That data cannot be used to challange the result (The way we state in Online contest of Dhaka regional).

-Shahriar

gvcormac
Problemsetter & Reviewer
Posts: 194
Joined: Fri Mar 15, 2002 2:00 am
Contact:

Post by gvcormac » Fri Apr 15, 2005 3:36 am

The finals and many regions have a policy of not releasing test data. The finals don't even release the final results, except in a severely summarized form.

I think the motivation behind secrecy is misguided. More controversy and ill-will arises from secrecy than from openness. The judges should have the courage to make their judgements in the open, even if the policy is that their decision is final. We know from other sports that bad calls occasionally occur - that's part of the nature of sport. Hiding them doesn't eliminate them, it doesn't make anybody feel any better, and it doesn't provide any incentive to improve judging accuracy.

Releasing "unofficial" data does not really solve the problem. As anybody who's used uva.es knows, unofficial data is often wrong. And I don't see how it meets the controversy-avoidance objectives of the judges. If a contestant's program works on the unofficial data, he or she will be likely to believe that the official judgement was wrong. If the unofficial data is wrong, the contest will be inclined to believe that the official data is also wrong.

I think that secrecy is anti-educational and contrary to the spirit of cooperation that this contest should engender. In my region (East Central North America) the problem sets, data, and judges' solutions are published. Errors have (rarely) occurred and the judges have resolved them in what I consider to be a very fair way.

If you are in a region that does not publish its data, please lobby the regional director to do so. I daresay these regions draw ideas from, and expect regional competitors to practices from, other regions and contests that have more open policies. This taking but giving nothing back simply isn't fair.

To his credit, Bill Poucher (ICPC executive director) will be requiring regions in future to provide data (official or unofficial) for the on-line judge. I do not know if this includes the finals, but I think it does not. As I have mentioned above, "unofficial" data has its drawbacks. And providing data for an on-line judge is a far cry from publishing the test data and model solutions.

shahriar_manzoor
System administrator & Problemsetter
Posts: 399
Joined: Sat Jan 12, 2002 2:00 am

Post by shahriar_manzoor » Fri Apr 15, 2005 5:32 am

gvcormac wrote:I think the motivation behind secrecy is misguided. More controversy and ill-will arises from secrecy than from openness. The judges should have the courage to make their judgements in the open, even if the policy is that their decision is final. We know from other sports that bad calls occasionally occur - that's part of the nature of sport. Hiding them doesn't eliminate them, it doesn't make anybody feel any better, and it doesn't provide any incentive to improve judging accuracy.
The reason I don't like to release the Judge data of Dhaka Regional is that I don't want the Online Judge to become useless to many. Also I would like to see the best Offline problem solver at the top of the ranklist of the Online Judge.
gvcormac wrote: Releasing "unofficial" data does not really solve the problem. As anybody who's used uva.es knows, unofficial data is often wrong. And I don't see how it meets the controversy-avoidance objectives of the judges. If a contestant's program works on the unofficial data, he or she will be likely to believe that the official judgement was wrong. If the unofficial data is wrong, the contest will be inclined to believe that the official data is also wrong.
I agree. But at the same time I have no idea how is the environment when someone is working with a sponsor who is spending huge amount of money for the contest.
gvcormac wrote: I think that secrecy is anti-educational and contrary to the spirit of cooperation that this contest should engender. In my region (East Central North America) the problem sets, data, and judges' solutions are published. Errors have (rarely) occurred and the judges have resolved them in what I consider to be a very fair way.
Publishing judge data has also a problem. Because then a contestant may get the habit to fix his mistakes only by using the judge data rather than thinking. But he has to think when he is participating in actual contest.
gvcormac wrote: If you are in a region that does not publish its data, please lobby the regional director to do so. I daresay these regions draw ideas from, and expect regional competitors to practices from, other regions and contests that have more open policies. This taking but giving nothing back simply isn't fair.
I am a little bit moderate in this issue. Persue the regional contest director to hold online contest, publish the data through an online judge etc. I think ICPC have formed a CII group which had a questionnaire this year to the RCDs which actually asked them to publish judge data, hold online contests. So I think situation will improve. And the people in CII are not united. For example I am against publishing judge solution, I am for using different sets of data for online judge and published data, someone else is for publishing everhting and so on.
gvcormac wrote: To his credit, Bill Poucher (ICPC executive director) will be requiring regions in future to provide data (official or unofficial) for the on-line judge. I do not know if this includes the finals, but I think it does not. As I have mentioned above, "unofficial" data has its drawbacks. And providing data for an on-line judge is a far cry from publishing the test data and model solutions.
Fortunately this applies for finals as well. So from next year all judge data should at least be published via online judge. If it is not please let us know (contestATacm.uva.es).

Also surprisingly publishing judge data does not always ensure correctness. In a Dhaka regional the judge data had a lot of mistakes (4 out of eight problems had minor or major mistakes). Me and many other people shouted a lot and so the authority to show their correctness published the judge data on the webpage. A coach used this data and mailed me his helpless situation on what to do when a student gets wrong answer using that judge data (Whether to give it correct verdict or wrong answer) as it had so many mistakes.

A contest top official of an indian region once was asked "Sir! R u going to publish the judge data?" and his reply was "We don't want to go to court and so we won't publish judge data.", so let's see what happens now :).

We have our differences but we all want the judges to make their best effort to make the problemset error free. It is a question of a students' career and when someone is playing with it, it is surely a criminal offense.

gvcormac
Problemsetter & Reviewer
Posts: 194
Joined: Fri Mar 15, 2002 2:00 am
Contact:

Post by gvcormac » Fri Apr 15, 2005 6:20 am

shahriar_manzoor wrote: The reason I don't like to release the Judge data of Dhaka Regional is that I don't want the Online Judge to become useless to many. Also I would like to see the best Offline problem solver at the top of the ranklist of the Online Judge.
I understand your point of view, but I think there are lots of problem sets other than those used for competition that fill this role. I do acknowledge that you and your region are sharing in this regard, and apologize for being harsh.
shahriar_manzoor wrote: I agree. But at the same time I have no idea how is the environment when someone is working with a sponsor who is spending huge amount of money for the contest.
I'm not sure why a sponsor should make any difference. Even in the highest levels of professional sport, they have to deal with possible judging errors. I don't think that secrecy helps at all. If somebody is going to sue, they'll just subpoena the judge data.
shahriar_manzoor wrote: Publishing judge data has also a problem. Because then a contestant may get the habit to fix his mistakes only by using the judge data rather than thinking. But he has to think when he is participating in actual contest.
I think you're more paternalistic than me. I think right now that lots of contestants are frustrated because they don't have pedagogic information, and they're lost as to why their programs fail. If some are unsophisticated enough that they are unable to discipline themselves so as to learn from solved problems, I think they aren't going to succeed anyway.
gvcormac wrote: If you are in a region that does not publish its data, please lobby the regional director to do so. I daresay these regions draw ideas from, and expect regional competitors to practices from, other regions and contests that have more open policies. This taking but giving nothing back simply isn't fair.
shahriar_manzoor wrote: I am a little bit moderate in this issue. Persue the regional contest director to hold online contest, publish the data through an online judge etc. I think ICPC have formed a CII group which had a questionnaire this year to the RCDs which actually asked them to publish judge data, hold online contests. So I think situation will improve. And the people in CII are not united. For example I am against publishing judge solution, I am for using different sets of data for online judge and published data, someone else is for publishing everhting and so on.
My harsh words were really directed against the regions that give nothing back; supplying it to the on-line judge does mitigate things a bit. However, I still think that data used for real ACM competitions should be public, for the reasons I've stated.
shahriar_manzoor wrote:
Fortunately this applies for finals as well. So from next year all judge data should at least be published via online judge. If it is not please let us know (contestATacm.uva.es).

Also surprisingly publishing judge data does not always ensure correctness. In a Dhaka regional the judge data had a lot of mistakes (4 out of eight problems had minor or major mistakes). Me and many other people shouted a lot and so the authority to show their correctness published the judge data on the webpage. A coach used this data and mailed me his helpless situation on what to do when a student gets wrong answer using that judge data (Whether to give it correct verdict or wrong answer) as it had so many mistakes.
Of course. There's nothing wrong with preparing a corrected set of data for the on-line judge. But they shouldn't do what ECNA did in '96, which is to silently correct the data (and not revise the standings) several weeks after the contest. Or publish without comment data that is known to be wrong.
shahriar_manzoor wrote:
A contest top official of an indian region once was asked "Sir! R u going to publish the judge data?" and his reply was "We don't want to go to court and so we won't publish judge data.", so let's see what happens now :).
I think this is totally specious. If someone sues, they simply subpoena the data and the organizers are in worse shape because they failed to mitigate damages.
shahriar_manzoor wrote: We have our differences but we all want the judges to make their best effort to make the problemset error free. It is a question of a students' career and when someone is playing with it, it is surely a criminal offense.
I think there are several objectives: to make the problem set and data error free, to mitigate the damage if and when errors occur, to educate contestants (and judges) as to the nature of testing, judging etc., and to provide practice material. I think our differences are not large but to the extent they exist, I think that I am placing more emphasis on the first three objectives, as opposed to providing practice material (of a particular sort).

Andrew Stankevich
New poster
Posts: 2
Joined: Wed Oct 08, 2003 10:42 pm
Contact:

Publishing tests after the contests

Post by Andrew Stankevich » Sat Apr 16, 2005 11:16 am

I am one of the judges at NEERC http://neerc.ifmo.ru and we always publish test data after the contest. Although marked unofficial, it is actulally always a real contest data and we are actively discussing the possibility to finally take a full responsibility for the data.

Starting this year, we also publish all the runs of all the contestants made during the contest. Actually, we never had any problems about that. The main key to ensure that all the test data is correct, at our point of view, is to check it. And we do, most of our model solutions make a thoroughful inspection of the test data before actually solving the problem, and assert that all limitations and assumptions about test data are indeed fulfilled. And we always have at least three different solutions for each problem.

About the idea that publishing the test data affects the online judge. At my point of view the problem archive at the online judge has always been and will probably always be the tool for practice, not for the competition. If the student wants to get to the top of the judge's rankist, it is fine, but if he wants to do it just for the fact to be on top, not to solve more problems, the student won't be successful at the real contest. And there are several boards, that publish accepted solutions for judge problems any way, so publishing test data, IMO, does not really matter too much.

On the other side, the ability to get test data for the contest gives much to the students to find their mistakes and to avoid them later. I was a contestant a couple of years ago, and still I do not know the reason my program for problem G on World Finals 2000 failed. Is that pedagogical? Does that help me to solve problems without test cases on contests better? Maybe learning what my mistake was could help me more?

Additionally, publishing test data allows running practice contests on really good problemsets which regional contests usually are, and using contest-prooved testsets for practicing helps to avoid thinking about possible errors in tests.

shahriar_manzoor
System administrator & Problemsetter
Posts: 399
Joined: Sat Jan 12, 2002 2:00 am

hmm

Post by shahriar_manzoor » Sat Apr 16, 2005 12:05 pm

To gord:
Actually I did not mind at all on what you said so I did not look for an apology :)

General:
The reason I think I differ with both of you is because you see your teams practice, you see the contestants looking for mistakes and so on and so you know their sufferings. If all the data is published then it becomes easier for coaches (is it a coincident that you are the coaches of two gold medalist teams :) this year) and contestants.

But the idea of Online Judge is to make the job of training a lot easier as well. As I know that in short time teams will be able to arrange virtual contests which will enable them to practice more and if someone is trying to look for mistakes they can seek help from message board.

Personally, for the Online Contests of UVa we will not publish judge data because then it creates all problems like zero second submissions, cheating etc. Not to mention that there are many people who does not participate in contests but just solve problems for fun and for a better rank in online judge. As a person associated with this site I would like to have some problems, which are unique to this site so that more people use this site and this site becomes popular. I have no motivations like you to do this work of problemsetting (my team will do well in contests) and my only motivation is to make this site popular so in that sense u can call me selfish.

About regionals: I have no objection if regionals publish their judge data and solutions but I am against making it compulsory. The idea of CII is to make an archive open for all to practice and as I have attended only one meeting of CII I may be lacking in knowlwdge. One goal of CII is to collect data from all regionals/finals and make an archive, to give facility of virtual contest and so on. So I don

gvcormac
Problemsetter & Reviewer
Posts: 194
Joined: Fri Mar 15, 2002 2:00 am
Contact:

Re: hmm

Post by gvcormac » Sat Apr 16, 2005 1:53 pm

shahriar_manzoor wrote: Personally, for the Online Contests of UVa we will not publish judge data because then it creates all problems like zero second submissions, cheating etc. Not to mention that there are many people who does not participate in contests but just solve problems for fun and for a better rank in online judge. As a person associated with this site I would like to have some problems, which are unique to this site so that more people use this site and this site becomes popular. I have no motivations like you to do this work of problemsetting (my team will do well in contests) and my only motivation is to make this site popular so in that sense u can call me selfish.
For the last while, uva.es has archived only its on-line contest problems. Most of these are kept private. I have no problem with this.
shahriar_manzoor wrote:
About regionals: I have no objection if regionals publish their judge data and solutions but I am against making it compulsory. The idea of CII is to make an archive open for all to practice
Also agreed. The regional directors should release contest data and I think they should be lobbied to do so. This issue is orthogonal to the CII.
shahriar_manzoor wrote:
So unless we create social awareness or make a big committee that will check correctness of all regional judge data, errors will occur. A coach who is not deeply related with his teams practice will not understand the pain of his team due to the error in judge data.
I think openness is the best way to nurture this social awareness. Prospective judges can see how good sets are made (multiple solutions, input specification checkers, etc.) and also the sources of errors (last-minute substitutions, non-independent solutions, language-specific assumptions, DOS/Unix/compiler incompatibilities, ...). Judges' efforts would also be on display, which may make them uncomfortable, but will incent them to be more diligent.
shahriar_manzoor wrote: A solution for reducing error can be to use 12 problemsets for 30 regional contest by ensuring that regionals in near time zones start at the same time and use the same problemset. But for this we need good coordination among regional contests but that cannot happen in a single day.
The regions have traditionally had a lot of autonomy. Some regions and super-regions have shared some problems (often with different data!) but not many. I'm not sure if imposing the problems from on high would be a good idea. Certainly not, if the process were to be as closed and generally heavy-handed as the world finals.

gvcormac
Problemsetter & Reviewer
Posts: 194
Joined: Fri Mar 15, 2002 2:00 am
Contact:

Re: Publishing tests after the contests

Post by gvcormac » Sat Apr 16, 2005 2:24 pm

Andrew Stankevich wrote:I am one of the judges at NEERC http://neerc.ifmo.ru and we always publish test data after the contest. Although marked unofficial, it is actulally always a real contest data and we are actively discussing the possibility to finally take a full responsibility for the data.
NEERC problems and data are always high quality, and we appreciate that. I have three somewhat unrelated follow-up points:

1. What compiler is used for the checker program (and the majority of solutions)? I suppose it is Delphi or some other Windows software. Every year I do manage to make it (sort of) work with Free Pascal or p2c on Linux, but it requires quite a bit of hacking and there are always some judge solutions that I can't make work.

2. I promised to put together a proposal to make the ACM contest more "spectator friendly." This proposal will have several components, starting with simply making sure that scoreboards appear in predictable places. More sophisticated things that might be done could involve commentary - viewing either submissions or work-in-progress, with experts explaining strategies, errors, likely outcomes. This could be done using multimedia or just a blog. For an analogy, think of television coverage of chess, billiards, or other sports that would be mind-numbing without real-time analysis.

I raise this issue here because of your comment about publishing contestant data. It seems to me that making the contest a good spectator sport would necessarily involve showing both contestant submissions (or work in progress) and judge solutions. I wonder if some teams would resist in the same way as some judges. Of course, the teams wouldn't have as much power, but this is not to say they should simply be trampled in this matter.

3. The finals allow "25 pages" of material to be brought to the contest. It might be very interesting to publish all this material. I know that Waterloo keeps their 25 pages pretty close to their chest, but I think that we wouldn't have a problem if everybody's book was revealed - perhaps even in real time to other teams during the contest, but in any event, later. I don't think this would have a profound effect on the top teams, but it would be another source of interest and pedagogy.

---

I would appreciate comments on any or all of the above issues.

shahriar_manzoor
System administrator & Problemsetter
Posts: 399
Joined: Sat Jan 12, 2002 2:00 am

Re: Publishing tests after the contests

Post by shahriar_manzoor » Sat Apr 16, 2005 3:08 pm

gvcormac wrote: NEERC problems and data are always high quality, and we appreciate that. I have three somewhat unrelated follow-up points:

1. What compiler is used for the checker program (and the majority of solutions)? I suppose it is Delphi or some other Windows software. Every year I do manage to make it (sort of) work with Free Pascal or p2c on Linux, but it requires quite a bit of hacking and there are always some judge solutions that I can't make work.
2. I promised to put together a proposal to make the ACM contest more "spectator friendly." This proposal will have several components, starting with simply making sure that scoreboards appear in predictable places. More sophisticated things that might be done could involve commentary - viewing either submissions or work-in-progress, with experts explaining strategies, errors, likely outcomes. This could be done using multimedia or just a blog. For an analogy, think of television coverage of chess, billiards, or other sports that would be mind-numbing without real-time analysis.
Nice concept. But I think we first need PC^2 to be more spectator friendly. Because most regions use PC^2 and also finals and I don't see that changing in near future. topcoder can be a good example of an expectator friendly contest. Also live video cameras can be a good idea which will show close up of teams (not their monitors) in a large screen.
gvcormac wrote: I raise this issue here because of your comment about publishing contestant data. It seems to me that making the contest a good spectator sport would necessarily involve showing both contestant submissions (or work in progress) and judge solutions. I wonder if some teams would resist in the same way as some judges. Of course, the teams wouldn't have as much power, but this is not to say they should simply be trampled in this matter.
If the large screen which is shown to the spectator is also viewable by the teams then showing other teams code should not be a good idea. Otherwise it is OK. As a judge when I write a solution I don't try to make my solution user firendly or neat or I don't add any comments. If I am informed that my solution will be public then I would surely make my solution compact, add some nice english comments, name variables in english rather than in my native language etc.
gvcormac wrote: 3. The finals allow "25 pages" of material to be brought to the contest. It might be very interesting to publish all this material. I know that Waterloo keeps their 25 pages pretty close to their chest, but I think that we wouldn't have a problem if everybody's book was revealed - perhaps even in real time to other teams during the contest, but in any event, later. I don't think this would have a profound effect on the top teams, but it would be another source of interest and pedagogy.
I am not a supporter of the 25 page idea, the open book idea was better. I would have even liked the contestants to have access to the internet (just if we could prevent them communicating with one another). During online contests teams cannot solve all problems in many contests with all these facilities so...
---
gvcormac wrote: I would appreciate comments on any or all of the above issues.
This thread is getting longer. I think it would be good idea to send this link to bill when this thread gets saturated.

shahriar_manzoor
System administrator & Problemsetter
Posts: 399
Joined: Sat Jan 12, 2002 2:00 am

hmm

Post by shahriar_manzoor » Sat Apr 16, 2005 3:17 pm

gvcormack wrote:
"Certainly not, if the process were to be as closed and generally heavy-handed as the world finals."

I think the world finals is a separate contest. But we have a european director, asian director, north american director so they should be able to coordinate.

I found the World Finals judging room a lot more comfortable than Dhaka regional judging room. Oh! May be I am a talkative judge, judges should be dumb.

User avatar
abishek
Experienced poster
Posts: 131
Joined: Mon Dec 15, 2003 5:41 am

Post by abishek » Mon Apr 18, 2005 3:59 am

http://www.ioi2004.org/html/competition_test.html

This is a link where problem/solution/test data for the IOI 2004 is available. I think IOI is also an as important competetion as the ACM and I think they are a setting a good example.

One thing to say is that the IOI is an one time competetion so its just like the ACM finals and there are no regionals except for countrywise selection of their candidates. May be the ACM can make it compulsory for the finals to release its solutions, test data and leave the regionals a voluntary option.

Also I think there is no code so no way of finding whether the judge solution had a bug!

The purpose of this request is not to find whether the judging was wrong (which is very possible and I am sure every one tries to avoid it) but to make life a bit easier for contestants who really want to practice, learn new ideas and see how the judges approached a problem, and like andrew said to find out what mistake he/she made in the contest.

shahriar_manzoor
System administrator & Problemsetter
Posts: 399
Joined: Sat Jan 12, 2002 2:00 am

hmm

Post by shahriar_manzoor » Mon Apr 18, 2005 4:31 am

When the problems will be added with Online Judge you will definitely find your mistake.

As far as I know the IOI problem setting does not involve so many people as ICPC does (as far as real working people r concerned). In IOI problemsetters or contributors are in many cases not the judges. So in that sense ICPC is more democratic than IOI.

gvcormac
Problemsetter & Reviewer
Posts: 194
Joined: Fri Mar 15, 2002 2:00 am
Contact:

Re: hmm

Post by gvcormac » Mon Apr 18, 2005 5:11 am

shahriar_manzoor wrote:When the problems will be added with Online Judge you will definitely find your mistake.

As far as I know the IOI problem setting does not involve so many people as ICPC does (as far as real working people r concerned). In IOI problemsetters or contributors are in many cases not the judges. So in that sense ICPC is more democratic than IOI.
IOI problems are solicited by the host country's Scientific Committee (SC), which considers the submissions and prepares a set of candidate problems. The International Scientific Committee (ISC) , with the SC, pares the problem set down and produces model solutions and test data. [I was elected to the ISC for three years starting this year.]

The final say on the problems is given by the General Assembly (GA), with one vote per country. Following this approval, non-English countries' delegates translate the problems. The contestants are kept in quarantine from the time the GA sees the problesm until after the competition. The whole process is repeated; that is, there are two separate competition days; each is handled by quarantining the competitors, approving the problems, translating the problems, holding the competition, and considering appeals.

Judging is automated, but the ISC and SC are both involved. The raw results (and test cases) are released to competitors, who may appeal their scores. Appeals are considered and final results are given.

So making the test cases available is very much part of the adjudication process.

Post Reply

Return to “Other words”