During peer-review, should I comment on the authors' messy code?

The name of the pictureThe name of the pictureThe name of the pictureClash Royale CLAN TAG#URR8PPP











up vote
99
down vote

favorite
5












I'm reviewing a paper in pure mathematics. A lot of results in the paper depend heavily on computer computations, and the authors have provided in the article a link to the Magma code they used for most of these computations. However, this code is almost impossible to understand due to the messy way it is written. For example, it does not use any indentation, and all variables are given names like 'aaa' or 'X' that do not give any information about their purpose in the program.



On the one hand, the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code (this is what I ended up doing). Also, the paper contains only a link to the code and not the actual code itself, so I'm not sure if the code is really in scope for the review. Moreover, hard-to-read code seems to not be uncommon in academia, and most people don't seem to mind. On the other hand, I think a small amount of work from the authors (who presumably do understand the code) would make this code a lot more usable for others, just by replacing some of the variable names with names that actually convey some meaning.



My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?










share|improve this question



















  • 37




    I don't see why the content being code makes it any different from a proof: if I refereed a paper with a proof which named variables inappropriately or didn't explain what was happening, I would ask the authors to improve it.
    – Henry
    Aug 29 at 0:27






  • 7




    If you ended up writing your own version of the code, possibly consider sending it to the author if you're willing to help them out? IDK if "collaborating" with the authors to that degree would step outside the bounds of what a reviewer should be doing, but from a practical standpoint it seems reasonable to me. (I'm an free-software geek, and not an academic).
    – Peter Cordes
    Aug 29 at 4:54






  • 7




    Relevant question on this site: Should I share my horrible software?
    – Dirk
    Aug 29 at 6:18






  • 7




    @DetlevCM Style is subjective, yes, but if your code doesn't have indents- sorry, but your code objectively sucks. The purpose of indentation is to make it easy to see the scope of various portions of your code (e.g. blocks, for loops, etc.). Not having that makes it much harder to read and decipher.
    – Jim Clay
    Aug 29 at 13:56






  • 5




    Do you think you could not have reviewed the paper without the code or is it an integral part? I do not mean if you with a good understanding and ability to reimplement it did not need it, but if you expect the average reader to be able to use the paper without the code.
    – allo
    Aug 29 at 14:15














up vote
99
down vote

favorite
5












I'm reviewing a paper in pure mathematics. A lot of results in the paper depend heavily on computer computations, and the authors have provided in the article a link to the Magma code they used for most of these computations. However, this code is almost impossible to understand due to the messy way it is written. For example, it does not use any indentation, and all variables are given names like 'aaa' or 'X' that do not give any information about their purpose in the program.



On the one hand, the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code (this is what I ended up doing). Also, the paper contains only a link to the code and not the actual code itself, so I'm not sure if the code is really in scope for the review. Moreover, hard-to-read code seems to not be uncommon in academia, and most people don't seem to mind. On the other hand, I think a small amount of work from the authors (who presumably do understand the code) would make this code a lot more usable for others, just by replacing some of the variable names with names that actually convey some meaning.



My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?










share|improve this question



















  • 37




    I don't see why the content being code makes it any different from a proof: if I refereed a paper with a proof which named variables inappropriately or didn't explain what was happening, I would ask the authors to improve it.
    – Henry
    Aug 29 at 0:27






  • 7




    If you ended up writing your own version of the code, possibly consider sending it to the author if you're willing to help them out? IDK if "collaborating" with the authors to that degree would step outside the bounds of what a reviewer should be doing, but from a practical standpoint it seems reasonable to me. (I'm an free-software geek, and not an academic).
    – Peter Cordes
    Aug 29 at 4:54






  • 7




    Relevant question on this site: Should I share my horrible software?
    – Dirk
    Aug 29 at 6:18






  • 7




    @DetlevCM Style is subjective, yes, but if your code doesn't have indents- sorry, but your code objectively sucks. The purpose of indentation is to make it easy to see the scope of various portions of your code (e.g. blocks, for loops, etc.). Not having that makes it much harder to read and decipher.
    – Jim Clay
    Aug 29 at 13:56






  • 5




    Do you think you could not have reviewed the paper without the code or is it an integral part? I do not mean if you with a good understanding and ability to reimplement it did not need it, but if you expect the average reader to be able to use the paper without the code.
    – allo
    Aug 29 at 14:15












up vote
99
down vote

favorite
5









up vote
99
down vote

favorite
5






5





I'm reviewing a paper in pure mathematics. A lot of results in the paper depend heavily on computer computations, and the authors have provided in the article a link to the Magma code they used for most of these computations. However, this code is almost impossible to understand due to the messy way it is written. For example, it does not use any indentation, and all variables are given names like 'aaa' or 'X' that do not give any information about their purpose in the program.



On the one hand, the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code (this is what I ended up doing). Also, the paper contains only a link to the code and not the actual code itself, so I'm not sure if the code is really in scope for the review. Moreover, hard-to-read code seems to not be uncommon in academia, and most people don't seem to mind. On the other hand, I think a small amount of work from the authors (who presumably do understand the code) would make this code a lot more usable for others, just by replacing some of the variable names with names that actually convey some meaning.



My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?










share|improve this question















I'm reviewing a paper in pure mathematics. A lot of results in the paper depend heavily on computer computations, and the authors have provided in the article a link to the Magma code they used for most of these computations. However, this code is almost impossible to understand due to the messy way it is written. For example, it does not use any indentation, and all variables are given names like 'aaa' or 'X' that do not give any information about their purpose in the program.



On the one hand, the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code (this is what I ended up doing). Also, the paper contains only a link to the code and not the actual code itself, so I'm not sure if the code is really in scope for the review. Moreover, hard-to-read code seems to not be uncommon in academia, and most people don't seem to mind. On the other hand, I think a small amount of work from the authors (who presumably do understand the code) would make this code a lot more usable for others, just by replacing some of the variable names with names that actually convey some meaning.



My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?







mathematics peer-review code






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Aug 29 at 6:26









Laurel

421410




421410










asked Aug 28 at 22:30









reviewer

486124




486124







  • 37




    I don't see why the content being code makes it any different from a proof: if I refereed a paper with a proof which named variables inappropriately or didn't explain what was happening, I would ask the authors to improve it.
    – Henry
    Aug 29 at 0:27






  • 7




    If you ended up writing your own version of the code, possibly consider sending it to the author if you're willing to help them out? IDK if "collaborating" with the authors to that degree would step outside the bounds of what a reviewer should be doing, but from a practical standpoint it seems reasonable to me. (I'm an free-software geek, and not an academic).
    – Peter Cordes
    Aug 29 at 4:54






  • 7




    Relevant question on this site: Should I share my horrible software?
    – Dirk
    Aug 29 at 6:18






  • 7




    @DetlevCM Style is subjective, yes, but if your code doesn't have indents- sorry, but your code objectively sucks. The purpose of indentation is to make it easy to see the scope of various portions of your code (e.g. blocks, for loops, etc.). Not having that makes it much harder to read and decipher.
    – Jim Clay
    Aug 29 at 13:56






  • 5




    Do you think you could not have reviewed the paper without the code or is it an integral part? I do not mean if you with a good understanding and ability to reimplement it did not need it, but if you expect the average reader to be able to use the paper without the code.
    – allo
    Aug 29 at 14:15












  • 37




    I don't see why the content being code makes it any different from a proof: if I refereed a paper with a proof which named variables inappropriately or didn't explain what was happening, I would ask the authors to improve it.
    – Henry
    Aug 29 at 0:27






  • 7




    If you ended up writing your own version of the code, possibly consider sending it to the author if you're willing to help them out? IDK if "collaborating" with the authors to that degree would step outside the bounds of what a reviewer should be doing, but from a practical standpoint it seems reasonable to me. (I'm an free-software geek, and not an academic).
    – Peter Cordes
    Aug 29 at 4:54






  • 7




    Relevant question on this site: Should I share my horrible software?
    – Dirk
    Aug 29 at 6:18






  • 7




    @DetlevCM Style is subjective, yes, but if your code doesn't have indents- sorry, but your code objectively sucks. The purpose of indentation is to make it easy to see the scope of various portions of your code (e.g. blocks, for loops, etc.). Not having that makes it much harder to read and decipher.
    – Jim Clay
    Aug 29 at 13:56






  • 5




    Do you think you could not have reviewed the paper without the code or is it an integral part? I do not mean if you with a good understanding and ability to reimplement it did not need it, but if you expect the average reader to be able to use the paper without the code.
    – allo
    Aug 29 at 14:15







37




37




I don't see why the content being code makes it any different from a proof: if I refereed a paper with a proof which named variables inappropriately or didn't explain what was happening, I would ask the authors to improve it.
– Henry
Aug 29 at 0:27




I don't see why the content being code makes it any different from a proof: if I refereed a paper with a proof which named variables inappropriately or didn't explain what was happening, I would ask the authors to improve it.
– Henry
Aug 29 at 0:27




7




7




If you ended up writing your own version of the code, possibly consider sending it to the author if you're willing to help them out? IDK if "collaborating" with the authors to that degree would step outside the bounds of what a reviewer should be doing, but from a practical standpoint it seems reasonable to me. (I'm an free-software geek, and not an academic).
– Peter Cordes
Aug 29 at 4:54




If you ended up writing your own version of the code, possibly consider sending it to the author if you're willing to help them out? IDK if "collaborating" with the authors to that degree would step outside the bounds of what a reviewer should be doing, but from a practical standpoint it seems reasonable to me. (I'm an free-software geek, and not an academic).
– Peter Cordes
Aug 29 at 4:54




7




7




Relevant question on this site: Should I share my horrible software?
– Dirk
Aug 29 at 6:18




Relevant question on this site: Should I share my horrible software?
– Dirk
Aug 29 at 6:18




7




7




@DetlevCM Style is subjective, yes, but if your code doesn't have indents- sorry, but your code objectively sucks. The purpose of indentation is to make it easy to see the scope of various portions of your code (e.g. blocks, for loops, etc.). Not having that makes it much harder to read and decipher.
– Jim Clay
Aug 29 at 13:56




@DetlevCM Style is subjective, yes, but if your code doesn't have indents- sorry, but your code objectively sucks. The purpose of indentation is to make it easy to see the scope of various portions of your code (e.g. blocks, for loops, etc.). Not having that makes it much harder to read and decipher.
– Jim Clay
Aug 29 at 13:56




5




5




Do you think you could not have reviewed the paper without the code or is it an integral part? I do not mean if you with a good understanding and ability to reimplement it did not need it, but if you expect the average reader to be able to use the paper without the code.
– allo
Aug 29 at 14:15




Do you think you could not have reviewed the paper without the code or is it an integral part? I do not mean if you with a good understanding and ability to reimplement it did not need it, but if you expect the average reader to be able to use the paper without the code.
– allo
Aug 29 at 14:15










9 Answers
9






active

oldest

votes

















up vote
96
down vote













If the authors have provided a link to their code as a reference, then it is appropriate to offer commentary, particularly if the article is based on the code.



However, I would recommend making the critique constructive: offer concrete suggestions for how to improve it rather than just saying it’s “messy” or “sloppy” and needs to be “cleaned up.”






share|improve this answer
















  • 2




    Doesn't this end up in practice just going to be a PR with the improved code? This looks like it's going to offload the work to write good code to reviewers who understand what that means. So people will just keep produce bad code and hope the reviewers will fix it for them. Sound like a wrong incentive system.
    – Hakaishin
    Aug 29 at 7:55






  • 1




    @Hakaishin Rejecting the paper is still an option.
    – Mast
    Aug 29 at 10:01






  • 26




    @Hakaishin I don't understand this reasoning. aeismail did not propose to clean up the code for the authors and send a pull request, he only suggested to provide actionable feedback rather than just saying the code is messy and needs to be improved. If we go by the reasoning that we can't give any helpful feedback at all, we could get rid of peer review and just do yes/no decisions on papers.
    – xLeitix
    Aug 29 at 10:35






  • 38




    @Hakaishin No. Constructive feedback doesn’t mean “fix the code for you.” The feedback could be as simple as “line up the code blocks better.” The point is that it should be actionable. A vague “fix it!” Helps no one. But yes, if the code is a major emphasis of the paper it’s fair for a reviewer to comment about it.
    – aeismail♦
    Aug 29 at 12:13






  • 2




    @aeismail Addendum to your comment: Constructive criticism, just like you said, isn't fixing the code for the person; it's just pointing out specific problems and, optionally, explaining why they are problems, rather than just saying "this is bad, fix it". That last is unconstructive because there's no way to know what is bad, and you might well make the same 'mistakes' again because you didn't know not to.
    – Nic Hartley
    Aug 30 at 22:59


















up vote
43
down vote













The code is within the scope of the review, and it is appropriate to review this and offer constructive suggestions in relation to its deficiencies. Now, bear in mind that the onus is on the author to satisfy reviewers of their argument, and if the argument depends on computer code that is so messy as to be unreadable, it is not incumbent on you to fix this for them. In this case, constructive advice might be limited to explaining why it is presently too hard to read (i.e., lack of indentation, unclear variable names, etc.), and this could reasonably lead to a recommendation to revise and resubmit. Try to be clear and comprehensive in describing why the code is presently difficult to read, so subsequent re-submissions can be expected to be up to scratch.



The best thing to do in these cases is to treat the computer code just like the prose in the paper. Just as with prose, the computer code needs to be clear and intelligible, relative to the standards for coding. If it is messy and unintelligible then it needs to be revised until it is clear. Reviewers do not shy away from rejecting papers when the prose is unintelligible, so it is perfectly reasonable to request that computer code be made intelligible.






share|improve this answer
















  • 3




    The code being a mess is probably not a valid reason for a major revision if the manuscript itself merits a more positive recommendation.
    – aeismail♦
    Aug 29 at 2:46






  • 22




    If it is too messy to be intelligible, and if the argument in the paper depends on it, then I would say you couldn't accept without changes. What would be more positive than revise and resubmit in that case?
    – Ben
    Aug 29 at 3:44






  • 6




    In this case, the OP verified the results without the code. This is a grey area. Personally I’d mention the issues but if everything else is fixed would leave the decision to the editor.
    – aeismail♦
    Aug 29 at 12:21


















up vote
16
down vote













Yes, you should comment and possibly more.



You've said it yourself:




A lot of results in the paper depend heavily on computer computations.




Well, the program code for computations is therefore part of the work you are reviewing. If the text of the paper was difficult to read, would you not consider that a weakness? Logically, therefore, the same is true for the code (even if it's to a slightly lesser extent).



Also, if the code is unreadable to you - maybe there are errors in it, despite the sound math underneath. And finally, if you can tell what the results should be without the code, then why even have the code?



So, if you feel the messiness does not preclude "parsing" the paper, then comment on it (and perhaps, if relevant, downgrade it from Strong Accept to Weak Accept, although perhaps that's too harsh - depends on the specifics.)



If you need to read the code to very the results, and you simply cannot, then that's a more serious problem. But before saying something like "Requires revision", consult with the journal editor / the program committee chair / etc.



Note: I'm a Computer Scientist, so my answer might be somewhat biased. On the other hand, I have written pure-theory paper with no code.






share|improve this answer





























    up vote
    10
    down vote













    Messy code affects reproducibility



    You tried to reproduce their results with the linked code, and were unable to do it. While you imply that you were ultimately able to develop your own code and replicate the results, I argue that badly written code affects reproducibility. In computer programming, this can be even more important, as programming languages don't necessarily have very long lives. Who knows if Magma or any other language will be common knowledge in 50 years.



    In the long view, reproducibility is the most important part of the scientific endeavour. Proof that doing a results in b, something that can be re-proven by anyone who cares to try, is an axiomatic building block upon which further scientific results can stand.



    If reproducibility is important, then there is nothing wrong with telling them to clean up their code. Frankly, if their code is as bad as you describe, it sounds like the authors will have trouble understanding their own work going back to it in a few years. In that case, but forcing them to learn a bit about writing nice code, you'd be doing them a favor.






    share|improve this answer
















    • 3




      "I wrote a program that found the answer to perpetual motion, but I wrote it in brainfuck and I was black out drunk at the time"
      – ivanivan
      Aug 30 at 3:19

















    up vote
    8
    down vote













    Let me touch briefly on an aspect that hasn't appeared on the existing answers.




    My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?




    Yes, you should comment on the code, but not only that: convince the authors that it is in their self-interest to fix those issues.



    Readable code is code that is easy to reuse. Reusable code is code that makes it easy to explore the mathematics presented in the paper. Explorable mathematics are more likely to have readers that find interesting extensions. Interesting extensions get published, and those publications cite the original code - and, moreover, provide some of the most valuable citations around.



    Making your code readable and reusable does not guarantee that this will happen, but if you publish unreadable code you're putting up an artificial barrier in front of a reader who might or might not go on to do further research based on your work, and if there are enough such barriers, that reader will just turn elsewhere. Making the code readable is a modest investment of time that results in a large improvement in the extensibility of the work.



    This putting-up-of-barriers, of course, isn't unique to code: unclear figures, tangled structure, messy grammar, missing lemmas, and all sorts of other issues can put in similar barriers, and your job as a reviewer includes pointing those out and helping the authors get rid of them. Code is no different - help them improve it!






    share|improve this answer



























      up vote
      4
      down vote













      I'm not in academia or a reviewer of articles/papers at this level (adjunct at tech school), but I do grade a lot of programming homework and the odd technical document sample, and I do software development to actually pay the bills.



      If the paper depended on output the code generated, then the code must be readable and understandable - otherwise, the code may not do what the author thinks it is doing and it is impossible for others to confirm w/o their own re-implementation. If such re-implementation is relatively trivial then it seems that the actual code isn't important, and so I would question why broken code for something that is easy to implement based on spec would be included or referenced in a scholarly paper.



      Given that you were able to verify using your own code implementation of his algorithm(s) I don't think that is the case, but it should be taken into consideration. Any decent IDE or even advanced text editor should be able to auto-format code and do project wide search/replace (refactoring). Kinda points to sheer laziness....






      share|improve this answer




















      • "Any decent IDE or even advanced text editor should be able to auto-format code" - On the contrary, it means anyone can format the code to their liking. Hence the lack of formatting is not a concern. Sure it's lazy, but it's just an inconvenience for the reader.
        – Nisarg Shah
        Aug 30 at 13:44






      • 3




        @NisargShah, an IDE can't give your variables meaningful names.
        – Wildcard
        Aug 30 at 21:26

















      up vote
      3
      down vote













      The authors provide a link in the article, so the code is either consider considered a reference or part of the research.
      Whatever the situation, this raises questions:



      1. Is the code archived? Practical ways to archive code include Zenodo or figshare. Code on a homepage is as good as no code at all.

      2. Is there a license to the code? If not, its status is not at all evident.

      As a reviewer, it is up to you to decide what to do. Possible actions include:



      1. Do not comment on the code.

      2. Comment on the code with what I would call the minimum: require for the code to be archived and licensed properly.

      3. Depending on the importance of the computer program in the research, require a minimal amount of readability and that the author provides some tests on the program (i.e. that the program provides known analytical answers if some parameter sets allow it, etc).

      Regarding archival, you can refer to the editorial information of the Journal of Open Research Software.






      share|improve this answer
















      • 1




        The "Journal of Open Research" does not seem exactly trustworthy given that the link to both Codeplex and Google code which have ceased operation quite some time ago. And lastly, one needs to pay for the publication which again sort of raises questions about the trustworthyness, especially given the outdated information and the apparent reliance on authors to host the code elsewhere... (Elsevier also has a "pay to publish" software journal - but they will/offer to host a copy of your code with the paper.)
        – DetlevCM
        Aug 29 at 11:27










      • Their publication fee is quite low in comparison to Elsevier and they originate from a scientific society (the Software Sustainability Institute, so I don't agree with your assessment. (disclaimer: I published with them). You are right with respect to the list of repositories, I will mention it to them.
        – Pierre de Buyl
        Aug 29 at 11:42










      • 325 Pounds vs 500 Euros isn't a huge difference. Now they may have a well reputed backer, but this should be properly communicated AND they should possibly keep their resources up to date. Google Code closed at the beginning of 2016 while Codeplex closed in December 2017. (Doing some more googling now, it seems like Ubiquity Press is normally a "recognised publisher" of open access articles - But the journal presentation does not exactly inspire confidence.)
        – DetlevCM
        Aug 29 at 12:51

















      up vote
      2
      down vote













      I'm a software engineer and I want to answer the question from that perspective. Most code is not readable in itself. You need to have comment to document data structures and specifications for subroutine calls. Academics are not software engineers, and I don't expect them to do a professional job in this respect. Still, it is certainly in order to comment on the quality of the software. Without looking at the actual code, I'm not sure I would comment that it is unreadable, because the article (that is according to testimony, sufficient to reproduce the code) is to be considered part of the program documentation. If the program uses short names, that are the same then in the paper, that is no problem. Missing indentation is not an indication of low quality, but many levels of indentation is.
      I would suggest that you express your feeling that you find it hard to read, but that you are no code expert either, and maybe have some software engineer look at it. It is a different skill set you know. That should take the edge off the comment.



      To top it off: I've done a good job in cleaning up code, that I didn't understand at the purpose level. You'll be surprised what an expert in a different field is capable of.



      Bottom line, the code is not of the essence, the quality of the code is incidental and it shouldn't affect your decision either way.






      share|improve this answer





























        up vote
        0
        down vote













        My understanding is the authors do not submit the code for publication, so the code does not seem to be subject to your review. The authors just give a link to their code. This begs a question: if they just had provided a reference to their work published elsewhere, would you have considered telling them that that work should be improved? One may argue that validity of their results hinges on the correctness of their code, but they could choose not to provide access to the code at all, as "the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code". You, as a reviewer, took the trouble to verify the results of their computation, thus going beyond and above your duty, but that means that their results are indeed reproducible.



        So I would think that you may mention that their code is lousy (while apparently correct), but that should not affect your recommendation on publishing / not publishing the paper.






        share|improve this answer
















        • 1




          If they provided a link to work elsewhere, and following that link lead to a page that was basically unreadable, it wouldn't be a bad idea to point that out to them.
          – RDFozz
          Aug 31 at 15:13










        • @RDFozz : That does not seem to contradict the conclusion in the last paragraph of my answer.
          – akhmeteli
          Aug 31 at 23:29










        • In my reading of the question, I see nothing that indicates the OP would recommend the paper not be published based on the state of the code (though, perhaps I'm missing something). I was focused on the first couple of sentences of your answer, and trying to point out the difference between a reference to a published work of some sort (which might require actually obtaining a physical copy of the work to look at it, and which is presumably in a format that would not allow simple revision), and a link to a web page (which generally is relatively easy to change/improve).
          – RDFozz
          Sep 1 at 0:42










        • @RDFozz : There may be some practical difference, but still the code is beyond the scope of review, and the authors may have all kinds of reasons not to touch the code.
          – akhmeteli
          Sep 1 at 1:27









        protected by Alexandros Aug 31 at 15:39



        Thank you for your interest in this question.
        Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).



        Would you like to answer one of these unanswered questions instead?














        9 Answers
        9






        active

        oldest

        votes








        9 Answers
        9






        active

        oldest

        votes









        active

        oldest

        votes






        active

        oldest

        votes








        up vote
        96
        down vote













        If the authors have provided a link to their code as a reference, then it is appropriate to offer commentary, particularly if the article is based on the code.



        However, I would recommend making the critique constructive: offer concrete suggestions for how to improve it rather than just saying it’s “messy” or “sloppy” and needs to be “cleaned up.”






        share|improve this answer
















        • 2




          Doesn't this end up in practice just going to be a PR with the improved code? This looks like it's going to offload the work to write good code to reviewers who understand what that means. So people will just keep produce bad code and hope the reviewers will fix it for them. Sound like a wrong incentive system.
          – Hakaishin
          Aug 29 at 7:55






        • 1




          @Hakaishin Rejecting the paper is still an option.
          – Mast
          Aug 29 at 10:01






        • 26




          @Hakaishin I don't understand this reasoning. aeismail did not propose to clean up the code for the authors and send a pull request, he only suggested to provide actionable feedback rather than just saying the code is messy and needs to be improved. If we go by the reasoning that we can't give any helpful feedback at all, we could get rid of peer review and just do yes/no decisions on papers.
          – xLeitix
          Aug 29 at 10:35






        • 38




          @Hakaishin No. Constructive feedback doesn’t mean “fix the code for you.” The feedback could be as simple as “line up the code blocks better.” The point is that it should be actionable. A vague “fix it!” Helps no one. But yes, if the code is a major emphasis of the paper it’s fair for a reviewer to comment about it.
          – aeismail♦
          Aug 29 at 12:13






        • 2




          @aeismail Addendum to your comment: Constructive criticism, just like you said, isn't fixing the code for the person; it's just pointing out specific problems and, optionally, explaining why they are problems, rather than just saying "this is bad, fix it". That last is unconstructive because there's no way to know what is bad, and you might well make the same 'mistakes' again because you didn't know not to.
          – Nic Hartley
          Aug 30 at 22:59















        up vote
        96
        down vote













        If the authors have provided a link to their code as a reference, then it is appropriate to offer commentary, particularly if the article is based on the code.



        However, I would recommend making the critique constructive: offer concrete suggestions for how to improve it rather than just saying it’s “messy” or “sloppy” and needs to be “cleaned up.”






        share|improve this answer
















        • 2




          Doesn't this end up in practice just going to be a PR with the improved code? This looks like it's going to offload the work to write good code to reviewers who understand what that means. So people will just keep produce bad code and hope the reviewers will fix it for them. Sound like a wrong incentive system.
          – Hakaishin
          Aug 29 at 7:55






        • 1




          @Hakaishin Rejecting the paper is still an option.
          – Mast
          Aug 29 at 10:01






        • 26




          @Hakaishin I don't understand this reasoning. aeismail did not propose to clean up the code for the authors and send a pull request, he only suggested to provide actionable feedback rather than just saying the code is messy and needs to be improved. If we go by the reasoning that we can't give any helpful feedback at all, we could get rid of peer review and just do yes/no decisions on papers.
          – xLeitix
          Aug 29 at 10:35






        • 38




          @Hakaishin No. Constructive feedback doesn’t mean “fix the code for you.” The feedback could be as simple as “line up the code blocks better.” The point is that it should be actionable. A vague “fix it!” Helps no one. But yes, if the code is a major emphasis of the paper it’s fair for a reviewer to comment about it.
          – aeismail♦
          Aug 29 at 12:13






        • 2




          @aeismail Addendum to your comment: Constructive criticism, just like you said, isn't fixing the code for the person; it's just pointing out specific problems and, optionally, explaining why they are problems, rather than just saying "this is bad, fix it". That last is unconstructive because there's no way to know what is bad, and you might well make the same 'mistakes' again because you didn't know not to.
          – Nic Hartley
          Aug 30 at 22:59













        up vote
        96
        down vote










        up vote
        96
        down vote









        If the authors have provided a link to their code as a reference, then it is appropriate to offer commentary, particularly if the article is based on the code.



        However, I would recommend making the critique constructive: offer concrete suggestions for how to improve it rather than just saying it’s “messy” or “sloppy” and needs to be “cleaned up.”






        share|improve this answer












        If the authors have provided a link to their code as a reference, then it is appropriate to offer commentary, particularly if the article is based on the code.



        However, I would recommend making the critique constructive: offer concrete suggestions for how to improve it rather than just saying it’s “messy” or “sloppy” and needs to be “cleaned up.”







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Aug 28 at 22:43









        aeismail♦

        155k29357682




        155k29357682







        • 2




          Doesn't this end up in practice just going to be a PR with the improved code? This looks like it's going to offload the work to write good code to reviewers who understand what that means. So people will just keep produce bad code and hope the reviewers will fix it for them. Sound like a wrong incentive system.
          – Hakaishin
          Aug 29 at 7:55






        • 1




          @Hakaishin Rejecting the paper is still an option.
          – Mast
          Aug 29 at 10:01






        • 26




          @Hakaishin I don't understand this reasoning. aeismail did not propose to clean up the code for the authors and send a pull request, he only suggested to provide actionable feedback rather than just saying the code is messy and needs to be improved. If we go by the reasoning that we can't give any helpful feedback at all, we could get rid of peer review and just do yes/no decisions on papers.
          – xLeitix
          Aug 29 at 10:35






        • 38




          @Hakaishin No. Constructive feedback doesn’t mean “fix the code for you.” The feedback could be as simple as “line up the code blocks better.” The point is that it should be actionable. A vague “fix it!” Helps no one. But yes, if the code is a major emphasis of the paper it’s fair for a reviewer to comment about it.
          – aeismail♦
          Aug 29 at 12:13






        • 2




          @aeismail Addendum to your comment: Constructive criticism, just like you said, isn't fixing the code for the person; it's just pointing out specific problems and, optionally, explaining why they are problems, rather than just saying "this is bad, fix it". That last is unconstructive because there's no way to know what is bad, and you might well make the same 'mistakes' again because you didn't know not to.
          – Nic Hartley
          Aug 30 at 22:59













        • 2




          Doesn't this end up in practice just going to be a PR with the improved code? This looks like it's going to offload the work to write good code to reviewers who understand what that means. So people will just keep produce bad code and hope the reviewers will fix it for them. Sound like a wrong incentive system.
          – Hakaishin
          Aug 29 at 7:55






        • 1




          @Hakaishin Rejecting the paper is still an option.
          – Mast
          Aug 29 at 10:01






        • 26




          @Hakaishin I don't understand this reasoning. aeismail did not propose to clean up the code for the authors and send a pull request, he only suggested to provide actionable feedback rather than just saying the code is messy and needs to be improved. If we go by the reasoning that we can't give any helpful feedback at all, we could get rid of peer review and just do yes/no decisions on papers.
          – xLeitix
          Aug 29 at 10:35






        • 38




          @Hakaishin No. Constructive feedback doesn’t mean “fix the code for you.” The feedback could be as simple as “line up the code blocks better.” The point is that it should be actionable. A vague “fix it!” Helps no one. But yes, if the code is a major emphasis of the paper it’s fair for a reviewer to comment about it.
          – aeismail♦
          Aug 29 at 12:13






        • 2




          @aeismail Addendum to your comment: Constructive criticism, just like you said, isn't fixing the code for the person; it's just pointing out specific problems and, optionally, explaining why they are problems, rather than just saying "this is bad, fix it". That last is unconstructive because there's no way to know what is bad, and you might well make the same 'mistakes' again because you didn't know not to.
          – Nic Hartley
          Aug 30 at 22:59








        2




        2




        Doesn't this end up in practice just going to be a PR with the improved code? This looks like it's going to offload the work to write good code to reviewers who understand what that means. So people will just keep produce bad code and hope the reviewers will fix it for them. Sound like a wrong incentive system.
        – Hakaishin
        Aug 29 at 7:55




        Doesn't this end up in practice just going to be a PR with the improved code? This looks like it's going to offload the work to write good code to reviewers who understand what that means. So people will just keep produce bad code and hope the reviewers will fix it for them. Sound like a wrong incentive system.
        – Hakaishin
        Aug 29 at 7:55




        1




        1




        @Hakaishin Rejecting the paper is still an option.
        – Mast
        Aug 29 at 10:01




        @Hakaishin Rejecting the paper is still an option.
        – Mast
        Aug 29 at 10:01




        26




        26




        @Hakaishin I don't understand this reasoning. aeismail did not propose to clean up the code for the authors and send a pull request, he only suggested to provide actionable feedback rather than just saying the code is messy and needs to be improved. If we go by the reasoning that we can't give any helpful feedback at all, we could get rid of peer review and just do yes/no decisions on papers.
        – xLeitix
        Aug 29 at 10:35




        @Hakaishin I don't understand this reasoning. aeismail did not propose to clean up the code for the authors and send a pull request, he only suggested to provide actionable feedback rather than just saying the code is messy and needs to be improved. If we go by the reasoning that we can't give any helpful feedback at all, we could get rid of peer review and just do yes/no decisions on papers.
        – xLeitix
        Aug 29 at 10:35




        38




        38




        @Hakaishin No. Constructive feedback doesn’t mean “fix the code for you.” The feedback could be as simple as “line up the code blocks better.” The point is that it should be actionable. A vague “fix it!” Helps no one. But yes, if the code is a major emphasis of the paper it’s fair for a reviewer to comment about it.
        – aeismail♦
        Aug 29 at 12:13




        @Hakaishin No. Constructive feedback doesn’t mean “fix the code for you.” The feedback could be as simple as “line up the code blocks better.” The point is that it should be actionable. A vague “fix it!” Helps no one. But yes, if the code is a major emphasis of the paper it’s fair for a reviewer to comment about it.
        – aeismail♦
        Aug 29 at 12:13




        2




        2




        @aeismail Addendum to your comment: Constructive criticism, just like you said, isn't fixing the code for the person; it's just pointing out specific problems and, optionally, explaining why they are problems, rather than just saying "this is bad, fix it". That last is unconstructive because there's no way to know what is bad, and you might well make the same 'mistakes' again because you didn't know not to.
        – Nic Hartley
        Aug 30 at 22:59





        @aeismail Addendum to your comment: Constructive criticism, just like you said, isn't fixing the code for the person; it's just pointing out specific problems and, optionally, explaining why they are problems, rather than just saying "this is bad, fix it". That last is unconstructive because there's no way to know what is bad, and you might well make the same 'mistakes' again because you didn't know not to.
        – Nic Hartley
        Aug 30 at 22:59











        up vote
        43
        down vote













        The code is within the scope of the review, and it is appropriate to review this and offer constructive suggestions in relation to its deficiencies. Now, bear in mind that the onus is on the author to satisfy reviewers of their argument, and if the argument depends on computer code that is so messy as to be unreadable, it is not incumbent on you to fix this for them. In this case, constructive advice might be limited to explaining why it is presently too hard to read (i.e., lack of indentation, unclear variable names, etc.), and this could reasonably lead to a recommendation to revise and resubmit. Try to be clear and comprehensive in describing why the code is presently difficult to read, so subsequent re-submissions can be expected to be up to scratch.



        The best thing to do in these cases is to treat the computer code just like the prose in the paper. Just as with prose, the computer code needs to be clear and intelligible, relative to the standards for coding. If it is messy and unintelligible then it needs to be revised until it is clear. Reviewers do not shy away from rejecting papers when the prose is unintelligible, so it is perfectly reasonable to request that computer code be made intelligible.






        share|improve this answer
















        • 3




          The code being a mess is probably not a valid reason for a major revision if the manuscript itself merits a more positive recommendation.
          – aeismail♦
          Aug 29 at 2:46






        • 22




          If it is too messy to be intelligible, and if the argument in the paper depends on it, then I would say you couldn't accept without changes. What would be more positive than revise and resubmit in that case?
          – Ben
          Aug 29 at 3:44






        • 6




          In this case, the OP verified the results without the code. This is a grey area. Personally I’d mention the issues but if everything else is fixed would leave the decision to the editor.
          – aeismail♦
          Aug 29 at 12:21















        up vote
        43
        down vote













        The code is within the scope of the review, and it is appropriate to review this and offer constructive suggestions in relation to its deficiencies. Now, bear in mind that the onus is on the author to satisfy reviewers of their argument, and if the argument depends on computer code that is so messy as to be unreadable, it is not incumbent on you to fix this for them. In this case, constructive advice might be limited to explaining why it is presently too hard to read (i.e., lack of indentation, unclear variable names, etc.), and this could reasonably lead to a recommendation to revise and resubmit. Try to be clear and comprehensive in describing why the code is presently difficult to read, so subsequent re-submissions can be expected to be up to scratch.



        The best thing to do in these cases is to treat the computer code just like the prose in the paper. Just as with prose, the computer code needs to be clear and intelligible, relative to the standards for coding. If it is messy and unintelligible then it needs to be revised until it is clear. Reviewers do not shy away from rejecting papers when the prose is unintelligible, so it is perfectly reasonable to request that computer code be made intelligible.






        share|improve this answer
















        • 3




          The code being a mess is probably not a valid reason for a major revision if the manuscript itself merits a more positive recommendation.
          – aeismail♦
          Aug 29 at 2:46






        • 22




          If it is too messy to be intelligible, and if the argument in the paper depends on it, then I would say you couldn't accept without changes. What would be more positive than revise and resubmit in that case?
          – Ben
          Aug 29 at 3:44






        • 6




          In this case, the OP verified the results without the code. This is a grey area. Personally I’d mention the issues but if everything else is fixed would leave the decision to the editor.
          – aeismail♦
          Aug 29 at 12:21













        up vote
        43
        down vote










        up vote
        43
        down vote









        The code is within the scope of the review, and it is appropriate to review this and offer constructive suggestions in relation to its deficiencies. Now, bear in mind that the onus is on the author to satisfy reviewers of their argument, and if the argument depends on computer code that is so messy as to be unreadable, it is not incumbent on you to fix this for them. In this case, constructive advice might be limited to explaining why it is presently too hard to read (i.e., lack of indentation, unclear variable names, etc.), and this could reasonably lead to a recommendation to revise and resubmit. Try to be clear and comprehensive in describing why the code is presently difficult to read, so subsequent re-submissions can be expected to be up to scratch.



        The best thing to do in these cases is to treat the computer code just like the prose in the paper. Just as with prose, the computer code needs to be clear and intelligible, relative to the standards for coding. If it is messy and unintelligible then it needs to be revised until it is clear. Reviewers do not shy away from rejecting papers when the prose is unintelligible, so it is perfectly reasonable to request that computer code be made intelligible.






        share|improve this answer












        The code is within the scope of the review, and it is appropriate to review this and offer constructive suggestions in relation to its deficiencies. Now, bear in mind that the onus is on the author to satisfy reviewers of their argument, and if the argument depends on computer code that is so messy as to be unreadable, it is not incumbent on you to fix this for them. In this case, constructive advice might be limited to explaining why it is presently too hard to read (i.e., lack of indentation, unclear variable names, etc.), and this could reasonably lead to a recommendation to revise and resubmit. Try to be clear and comprehensive in describing why the code is presently difficult to read, so subsequent re-submissions can be expected to be up to scratch.



        The best thing to do in these cases is to treat the computer code just like the prose in the paper. Just as with prose, the computer code needs to be clear and intelligible, relative to the standards for coding. If it is messy and unintelligible then it needs to be revised until it is clear. Reviewers do not shy away from rejecting papers when the prose is unintelligible, so it is perfectly reasonable to request that computer code be made intelligible.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered Aug 29 at 1:06









        Ben

        7,6671942




        7,6671942







        • 3




          The code being a mess is probably not a valid reason for a major revision if the manuscript itself merits a more positive recommendation.
          – aeismail♦
          Aug 29 at 2:46






        • 22




          If it is too messy to be intelligible, and if the argument in the paper depends on it, then I would say you couldn't accept without changes. What would be more positive than revise and resubmit in that case?
          – Ben
          Aug 29 at 3:44






        • 6




          In this case, the OP verified the results without the code. This is a grey area. Personally I’d mention the issues but if everything else is fixed would leave the decision to the editor.
          – aeismail♦
          Aug 29 at 12:21













        • 3




          The code being a mess is probably not a valid reason for a major revision if the manuscript itself merits a more positive recommendation.
          – aeismail♦
          Aug 29 at 2:46






        • 22




          If it is too messy to be intelligible, and if the argument in the paper depends on it, then I would say you couldn't accept without changes. What would be more positive than revise and resubmit in that case?
          – Ben
          Aug 29 at 3:44






        • 6




          In this case, the OP verified the results without the code. This is a grey area. Personally I’d mention the issues but if everything else is fixed would leave the decision to the editor.
          – aeismail♦
          Aug 29 at 12:21








        3




        3




        The code being a mess is probably not a valid reason for a major revision if the manuscript itself merits a more positive recommendation.
        – aeismail♦
        Aug 29 at 2:46




        The code being a mess is probably not a valid reason for a major revision if the manuscript itself merits a more positive recommendation.
        – aeismail♦
        Aug 29 at 2:46




        22




        22




        If it is too messy to be intelligible, and if the argument in the paper depends on it, then I would say you couldn't accept without changes. What would be more positive than revise and resubmit in that case?
        – Ben
        Aug 29 at 3:44




        If it is too messy to be intelligible, and if the argument in the paper depends on it, then I would say you couldn't accept without changes. What would be more positive than revise and resubmit in that case?
        – Ben
        Aug 29 at 3:44




        6




        6




        In this case, the OP verified the results without the code. This is a grey area. Personally I’d mention the issues but if everything else is fixed would leave the decision to the editor.
        – aeismail♦
        Aug 29 at 12:21





        In this case, the OP verified the results without the code. This is a grey area. Personally I’d mention the issues but if everything else is fixed would leave the decision to the editor.
        – aeismail♦
        Aug 29 at 12:21











        up vote
        16
        down vote













        Yes, you should comment and possibly more.



        You've said it yourself:




        A lot of results in the paper depend heavily on computer computations.




        Well, the program code for computations is therefore part of the work you are reviewing. If the text of the paper was difficult to read, would you not consider that a weakness? Logically, therefore, the same is true for the code (even if it's to a slightly lesser extent).



        Also, if the code is unreadable to you - maybe there are errors in it, despite the sound math underneath. And finally, if you can tell what the results should be without the code, then why even have the code?



        So, if you feel the messiness does not preclude "parsing" the paper, then comment on it (and perhaps, if relevant, downgrade it from Strong Accept to Weak Accept, although perhaps that's too harsh - depends on the specifics.)



        If you need to read the code to very the results, and you simply cannot, then that's a more serious problem. But before saying something like "Requires revision", consult with the journal editor / the program committee chair / etc.



        Note: I'm a Computer Scientist, so my answer might be somewhat biased. On the other hand, I have written pure-theory paper with no code.






        share|improve this answer


























          up vote
          16
          down vote













          Yes, you should comment and possibly more.



          You've said it yourself:




          A lot of results in the paper depend heavily on computer computations.




          Well, the program code for computations is therefore part of the work you are reviewing. If the text of the paper was difficult to read, would you not consider that a weakness? Logically, therefore, the same is true for the code (even if it's to a slightly lesser extent).



          Also, if the code is unreadable to you - maybe there are errors in it, despite the sound math underneath. And finally, if you can tell what the results should be without the code, then why even have the code?



          So, if you feel the messiness does not preclude "parsing" the paper, then comment on it (and perhaps, if relevant, downgrade it from Strong Accept to Weak Accept, although perhaps that's too harsh - depends on the specifics.)



          If you need to read the code to very the results, and you simply cannot, then that's a more serious problem. But before saying something like "Requires revision", consult with the journal editor / the program committee chair / etc.



          Note: I'm a Computer Scientist, so my answer might be somewhat biased. On the other hand, I have written pure-theory paper with no code.






          share|improve this answer
























            up vote
            16
            down vote










            up vote
            16
            down vote









            Yes, you should comment and possibly more.



            You've said it yourself:




            A lot of results in the paper depend heavily on computer computations.




            Well, the program code for computations is therefore part of the work you are reviewing. If the text of the paper was difficult to read, would you not consider that a weakness? Logically, therefore, the same is true for the code (even if it's to a slightly lesser extent).



            Also, if the code is unreadable to you - maybe there are errors in it, despite the sound math underneath. And finally, if you can tell what the results should be without the code, then why even have the code?



            So, if you feel the messiness does not preclude "parsing" the paper, then comment on it (and perhaps, if relevant, downgrade it from Strong Accept to Weak Accept, although perhaps that's too harsh - depends on the specifics.)



            If you need to read the code to very the results, and you simply cannot, then that's a more serious problem. But before saying something like "Requires revision", consult with the journal editor / the program committee chair / etc.



            Note: I'm a Computer Scientist, so my answer might be somewhat biased. On the other hand, I have written pure-theory paper with no code.






            share|improve this answer














            Yes, you should comment and possibly more.



            You've said it yourself:




            A lot of results in the paper depend heavily on computer computations.




            Well, the program code for computations is therefore part of the work you are reviewing. If the text of the paper was difficult to read, would you not consider that a weakness? Logically, therefore, the same is true for the code (even if it's to a slightly lesser extent).



            Also, if the code is unreadable to you - maybe there are errors in it, despite the sound math underneath. And finally, if you can tell what the results should be without the code, then why even have the code?



            So, if you feel the messiness does not preclude "parsing" the paper, then comment on it (and perhaps, if relevant, downgrade it from Strong Accept to Weak Accept, although perhaps that's too harsh - depends on the specifics.)



            If you need to read the code to very the results, and you simply cannot, then that's a more serious problem. But before saying something like "Requires revision", consult with the journal editor / the program committee chair / etc.



            Note: I'm a Computer Scientist, so my answer might be somewhat biased. On the other hand, I have written pure-theory paper with no code.







            share|improve this answer














            share|improve this answer



            share|improve this answer








            edited Aug 31 at 11:39

























            answered Aug 29 at 10:28









            einpoklum

            20.6k132118




            20.6k132118




















                up vote
                10
                down vote













                Messy code affects reproducibility



                You tried to reproduce their results with the linked code, and were unable to do it. While you imply that you were ultimately able to develop your own code and replicate the results, I argue that badly written code affects reproducibility. In computer programming, this can be even more important, as programming languages don't necessarily have very long lives. Who knows if Magma or any other language will be common knowledge in 50 years.



                In the long view, reproducibility is the most important part of the scientific endeavour. Proof that doing a results in b, something that can be re-proven by anyone who cares to try, is an axiomatic building block upon which further scientific results can stand.



                If reproducibility is important, then there is nothing wrong with telling them to clean up their code. Frankly, if their code is as bad as you describe, it sounds like the authors will have trouble understanding their own work going back to it in a few years. In that case, but forcing them to learn a bit about writing nice code, you'd be doing them a favor.






                share|improve this answer
















                • 3




                  "I wrote a program that found the answer to perpetual motion, but I wrote it in brainfuck and I was black out drunk at the time"
                  – ivanivan
                  Aug 30 at 3:19














                up vote
                10
                down vote













                Messy code affects reproducibility



                You tried to reproduce their results with the linked code, and were unable to do it. While you imply that you were ultimately able to develop your own code and replicate the results, I argue that badly written code affects reproducibility. In computer programming, this can be even more important, as programming languages don't necessarily have very long lives. Who knows if Magma or any other language will be common knowledge in 50 years.



                In the long view, reproducibility is the most important part of the scientific endeavour. Proof that doing a results in b, something that can be re-proven by anyone who cares to try, is an axiomatic building block upon which further scientific results can stand.



                If reproducibility is important, then there is nothing wrong with telling them to clean up their code. Frankly, if their code is as bad as you describe, it sounds like the authors will have trouble understanding their own work going back to it in a few years. In that case, but forcing them to learn a bit about writing nice code, you'd be doing them a favor.






                share|improve this answer
















                • 3




                  "I wrote a program that found the answer to perpetual motion, but I wrote it in brainfuck and I was black out drunk at the time"
                  – ivanivan
                  Aug 30 at 3:19












                up vote
                10
                down vote










                up vote
                10
                down vote









                Messy code affects reproducibility



                You tried to reproduce their results with the linked code, and were unable to do it. While you imply that you were ultimately able to develop your own code and replicate the results, I argue that badly written code affects reproducibility. In computer programming, this can be even more important, as programming languages don't necessarily have very long lives. Who knows if Magma or any other language will be common knowledge in 50 years.



                In the long view, reproducibility is the most important part of the scientific endeavour. Proof that doing a results in b, something that can be re-proven by anyone who cares to try, is an axiomatic building block upon which further scientific results can stand.



                If reproducibility is important, then there is nothing wrong with telling them to clean up their code. Frankly, if their code is as bad as you describe, it sounds like the authors will have trouble understanding their own work going back to it in a few years. In that case, but forcing them to learn a bit about writing nice code, you'd be doing them a favor.






                share|improve this answer












                Messy code affects reproducibility



                You tried to reproduce their results with the linked code, and were unable to do it. While you imply that you were ultimately able to develop your own code and replicate the results, I argue that badly written code affects reproducibility. In computer programming, this can be even more important, as programming languages don't necessarily have very long lives. Who knows if Magma or any other language will be common knowledge in 50 years.



                In the long view, reproducibility is the most important part of the scientific endeavour. Proof that doing a results in b, something that can be re-proven by anyone who cares to try, is an axiomatic building block upon which further scientific results can stand.



                If reproducibility is important, then there is nothing wrong with telling them to clean up their code. Frankly, if their code is as bad as you describe, it sounds like the authors will have trouble understanding their own work going back to it in a few years. In that case, but forcing them to learn a bit about writing nice code, you'd be doing them a favor.







                share|improve this answer












                share|improve this answer



                share|improve this answer










                answered Aug 29 at 12:47









                kingledion

                1,032239




                1,032239







                • 3




                  "I wrote a program that found the answer to perpetual motion, but I wrote it in brainfuck and I was black out drunk at the time"
                  – ivanivan
                  Aug 30 at 3:19












                • 3




                  "I wrote a program that found the answer to perpetual motion, but I wrote it in brainfuck and I was black out drunk at the time"
                  – ivanivan
                  Aug 30 at 3:19







                3




                3




                "I wrote a program that found the answer to perpetual motion, but I wrote it in brainfuck and I was black out drunk at the time"
                – ivanivan
                Aug 30 at 3:19




                "I wrote a program that found the answer to perpetual motion, but I wrote it in brainfuck and I was black out drunk at the time"
                – ivanivan
                Aug 30 at 3:19










                up vote
                8
                down vote













                Let me touch briefly on an aspect that hasn't appeared on the existing answers.




                My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?




                Yes, you should comment on the code, but not only that: convince the authors that it is in their self-interest to fix those issues.



                Readable code is code that is easy to reuse. Reusable code is code that makes it easy to explore the mathematics presented in the paper. Explorable mathematics are more likely to have readers that find interesting extensions. Interesting extensions get published, and those publications cite the original code - and, moreover, provide some of the most valuable citations around.



                Making your code readable and reusable does not guarantee that this will happen, but if you publish unreadable code you're putting up an artificial barrier in front of a reader who might or might not go on to do further research based on your work, and if there are enough such barriers, that reader will just turn elsewhere. Making the code readable is a modest investment of time that results in a large improvement in the extensibility of the work.



                This putting-up-of-barriers, of course, isn't unique to code: unclear figures, tangled structure, messy grammar, missing lemmas, and all sorts of other issues can put in similar barriers, and your job as a reviewer includes pointing those out and helping the authors get rid of them. Code is no different - help them improve it!






                share|improve this answer
























                  up vote
                  8
                  down vote













                  Let me touch briefly on an aspect that hasn't appeared on the existing answers.




                  My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?




                  Yes, you should comment on the code, but not only that: convince the authors that it is in their self-interest to fix those issues.



                  Readable code is code that is easy to reuse. Reusable code is code that makes it easy to explore the mathematics presented in the paper. Explorable mathematics are more likely to have readers that find interesting extensions. Interesting extensions get published, and those publications cite the original code - and, moreover, provide some of the most valuable citations around.



                  Making your code readable and reusable does not guarantee that this will happen, but if you publish unreadable code you're putting up an artificial barrier in front of a reader who might or might not go on to do further research based on your work, and if there are enough such barriers, that reader will just turn elsewhere. Making the code readable is a modest investment of time that results in a large improvement in the extensibility of the work.



                  This putting-up-of-barriers, of course, isn't unique to code: unclear figures, tangled structure, messy grammar, missing lemmas, and all sorts of other issues can put in similar barriers, and your job as a reviewer includes pointing those out and helping the authors get rid of them. Code is no different - help them improve it!






                  share|improve this answer






















                    up vote
                    8
                    down vote










                    up vote
                    8
                    down vote









                    Let me touch briefly on an aspect that hasn't appeared on the existing answers.




                    My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?




                    Yes, you should comment on the code, but not only that: convince the authors that it is in their self-interest to fix those issues.



                    Readable code is code that is easy to reuse. Reusable code is code that makes it easy to explore the mathematics presented in the paper. Explorable mathematics are more likely to have readers that find interesting extensions. Interesting extensions get published, and those publications cite the original code - and, moreover, provide some of the most valuable citations around.



                    Making your code readable and reusable does not guarantee that this will happen, but if you publish unreadable code you're putting up an artificial barrier in front of a reader who might or might not go on to do further research based on your work, and if there are enough such barriers, that reader will just turn elsewhere. Making the code readable is a modest investment of time that results in a large improvement in the extensibility of the work.



                    This putting-up-of-barriers, of course, isn't unique to code: unclear figures, tangled structure, messy grammar, missing lemmas, and all sorts of other issues can put in similar barriers, and your job as a reviewer includes pointing those out and helping the authors get rid of them. Code is no different - help them improve it!






                    share|improve this answer












                    Let me touch briefly on an aspect that hasn't appeared on the existing answers.




                    My question is, is it reasonable for me to tell the authors that their code is needlessly difficult to understand and should be improved?




                    Yes, you should comment on the code, but not only that: convince the authors that it is in their self-interest to fix those issues.



                    Readable code is code that is easy to reuse. Reusable code is code that makes it easy to explore the mathematics presented in the paper. Explorable mathematics are more likely to have readers that find interesting extensions. Interesting extensions get published, and those publications cite the original code - and, moreover, provide some of the most valuable citations around.



                    Making your code readable and reusable does not guarantee that this will happen, but if you publish unreadable code you're putting up an artificial barrier in front of a reader who might or might not go on to do further research based on your work, and if there are enough such barriers, that reader will just turn elsewhere. Making the code readable is a modest investment of time that results in a large improvement in the extensibility of the work.



                    This putting-up-of-barriers, of course, isn't unique to code: unclear figures, tangled structure, messy grammar, missing lemmas, and all sorts of other issues can put in similar barriers, and your job as a reviewer includes pointing those out and helping the authors get rid of them. Code is no different - help them improve it!







                    share|improve this answer












                    share|improve this answer



                    share|improve this answer










                    answered Aug 29 at 12:45









                    E.P.

                    4,6382453




                    4,6382453




















                        up vote
                        4
                        down vote













                        I'm not in academia or a reviewer of articles/papers at this level (adjunct at tech school), but I do grade a lot of programming homework and the odd technical document sample, and I do software development to actually pay the bills.



                        If the paper depended on output the code generated, then the code must be readable and understandable - otherwise, the code may not do what the author thinks it is doing and it is impossible for others to confirm w/o their own re-implementation. If such re-implementation is relatively trivial then it seems that the actual code isn't important, and so I would question why broken code for something that is easy to implement based on spec would be included or referenced in a scholarly paper.



                        Given that you were able to verify using your own code implementation of his algorithm(s) I don't think that is the case, but it should be taken into consideration. Any decent IDE or even advanced text editor should be able to auto-format code and do project wide search/replace (refactoring). Kinda points to sheer laziness....






                        share|improve this answer




















                        • "Any decent IDE or even advanced text editor should be able to auto-format code" - On the contrary, it means anyone can format the code to their liking. Hence the lack of formatting is not a concern. Sure it's lazy, but it's just an inconvenience for the reader.
                          – Nisarg Shah
                          Aug 30 at 13:44






                        • 3




                          @NisargShah, an IDE can't give your variables meaningful names.
                          – Wildcard
                          Aug 30 at 21:26














                        up vote
                        4
                        down vote













                        I'm not in academia or a reviewer of articles/papers at this level (adjunct at tech school), but I do grade a lot of programming homework and the odd technical document sample, and I do software development to actually pay the bills.



                        If the paper depended on output the code generated, then the code must be readable and understandable - otherwise, the code may not do what the author thinks it is doing and it is impossible for others to confirm w/o their own re-implementation. If such re-implementation is relatively trivial then it seems that the actual code isn't important, and so I would question why broken code for something that is easy to implement based on spec would be included or referenced in a scholarly paper.



                        Given that you were able to verify using your own code implementation of his algorithm(s) I don't think that is the case, but it should be taken into consideration. Any decent IDE or even advanced text editor should be able to auto-format code and do project wide search/replace (refactoring). Kinda points to sheer laziness....






                        share|improve this answer




















                        • "Any decent IDE or even advanced text editor should be able to auto-format code" - On the contrary, it means anyone can format the code to their liking. Hence the lack of formatting is not a concern. Sure it's lazy, but it's just an inconvenience for the reader.
                          – Nisarg Shah
                          Aug 30 at 13:44






                        • 3




                          @NisargShah, an IDE can't give your variables meaningful names.
                          – Wildcard
                          Aug 30 at 21:26












                        up vote
                        4
                        down vote










                        up vote
                        4
                        down vote









                        I'm not in academia or a reviewer of articles/papers at this level (adjunct at tech school), but I do grade a lot of programming homework and the odd technical document sample, and I do software development to actually pay the bills.



                        If the paper depended on output the code generated, then the code must be readable and understandable - otherwise, the code may not do what the author thinks it is doing and it is impossible for others to confirm w/o their own re-implementation. If such re-implementation is relatively trivial then it seems that the actual code isn't important, and so I would question why broken code for something that is easy to implement based on spec would be included or referenced in a scholarly paper.



                        Given that you were able to verify using your own code implementation of his algorithm(s) I don't think that is the case, but it should be taken into consideration. Any decent IDE or even advanced text editor should be able to auto-format code and do project wide search/replace (refactoring). Kinda points to sheer laziness....






                        share|improve this answer












                        I'm not in academia or a reviewer of articles/papers at this level (adjunct at tech school), but I do grade a lot of programming homework and the odd technical document sample, and I do software development to actually pay the bills.



                        If the paper depended on output the code generated, then the code must be readable and understandable - otherwise, the code may not do what the author thinks it is doing and it is impossible for others to confirm w/o their own re-implementation. If such re-implementation is relatively trivial then it seems that the actual code isn't important, and so I would question why broken code for something that is easy to implement based on spec would be included or referenced in a scholarly paper.



                        Given that you were able to verify using your own code implementation of his algorithm(s) I don't think that is the case, but it should be taken into consideration. Any decent IDE or even advanced text editor should be able to auto-format code and do project wide search/replace (refactoring). Kinda points to sheer laziness....







                        share|improve this answer












                        share|improve this answer



                        share|improve this answer










                        answered Aug 30 at 3:17









                        ivanivan

                        32715




                        32715











                        • "Any decent IDE or even advanced text editor should be able to auto-format code" - On the contrary, it means anyone can format the code to their liking. Hence the lack of formatting is not a concern. Sure it's lazy, but it's just an inconvenience for the reader.
                          – Nisarg Shah
                          Aug 30 at 13:44






                        • 3




                          @NisargShah, an IDE can't give your variables meaningful names.
                          – Wildcard
                          Aug 30 at 21:26
















                        • "Any decent IDE or even advanced text editor should be able to auto-format code" - On the contrary, it means anyone can format the code to their liking. Hence the lack of formatting is not a concern. Sure it's lazy, but it's just an inconvenience for the reader.
                          – Nisarg Shah
                          Aug 30 at 13:44






                        • 3




                          @NisargShah, an IDE can't give your variables meaningful names.
                          – Wildcard
                          Aug 30 at 21:26















                        "Any decent IDE or even advanced text editor should be able to auto-format code" - On the contrary, it means anyone can format the code to their liking. Hence the lack of formatting is not a concern. Sure it's lazy, but it's just an inconvenience for the reader.
                        – Nisarg Shah
                        Aug 30 at 13:44




                        "Any decent IDE or even advanced text editor should be able to auto-format code" - On the contrary, it means anyone can format the code to their liking. Hence the lack of formatting is not a concern. Sure it's lazy, but it's just an inconvenience for the reader.
                        – Nisarg Shah
                        Aug 30 at 13:44




                        3




                        3




                        @NisargShah, an IDE can't give your variables meaningful names.
                        – Wildcard
                        Aug 30 at 21:26




                        @NisargShah, an IDE can't give your variables meaningful names.
                        – Wildcard
                        Aug 30 at 21:26










                        up vote
                        3
                        down vote













                        The authors provide a link in the article, so the code is either consider considered a reference or part of the research.
                        Whatever the situation, this raises questions:



                        1. Is the code archived? Practical ways to archive code include Zenodo or figshare. Code on a homepage is as good as no code at all.

                        2. Is there a license to the code? If not, its status is not at all evident.

                        As a reviewer, it is up to you to decide what to do. Possible actions include:



                        1. Do not comment on the code.

                        2. Comment on the code with what I would call the minimum: require for the code to be archived and licensed properly.

                        3. Depending on the importance of the computer program in the research, require a minimal amount of readability and that the author provides some tests on the program (i.e. that the program provides known analytical answers if some parameter sets allow it, etc).

                        Regarding archival, you can refer to the editorial information of the Journal of Open Research Software.






                        share|improve this answer
















                        • 1




                          The "Journal of Open Research" does not seem exactly trustworthy given that the link to both Codeplex and Google code which have ceased operation quite some time ago. And lastly, one needs to pay for the publication which again sort of raises questions about the trustworthyness, especially given the outdated information and the apparent reliance on authors to host the code elsewhere... (Elsevier also has a "pay to publish" software journal - but they will/offer to host a copy of your code with the paper.)
                          – DetlevCM
                          Aug 29 at 11:27










                        • Their publication fee is quite low in comparison to Elsevier and they originate from a scientific society (the Software Sustainability Institute, so I don't agree with your assessment. (disclaimer: I published with them). You are right with respect to the list of repositories, I will mention it to them.
                          – Pierre de Buyl
                          Aug 29 at 11:42










                        • 325 Pounds vs 500 Euros isn't a huge difference. Now they may have a well reputed backer, but this should be properly communicated AND they should possibly keep their resources up to date. Google Code closed at the beginning of 2016 while Codeplex closed in December 2017. (Doing some more googling now, it seems like Ubiquity Press is normally a "recognised publisher" of open access articles - But the journal presentation does not exactly inspire confidence.)
                          – DetlevCM
                          Aug 29 at 12:51














                        up vote
                        3
                        down vote













                        The authors provide a link in the article, so the code is either consider considered a reference or part of the research.
                        Whatever the situation, this raises questions:



                        1. Is the code archived? Practical ways to archive code include Zenodo or figshare. Code on a homepage is as good as no code at all.

                        2. Is there a license to the code? If not, its status is not at all evident.

                        As a reviewer, it is up to you to decide what to do. Possible actions include:



                        1. Do not comment on the code.

                        2. Comment on the code with what I would call the minimum: require for the code to be archived and licensed properly.

                        3. Depending on the importance of the computer program in the research, require a minimal amount of readability and that the author provides some tests on the program (i.e. that the program provides known analytical answers if some parameter sets allow it, etc).

                        Regarding archival, you can refer to the editorial information of the Journal of Open Research Software.






                        share|improve this answer
















                        • 1




                          The "Journal of Open Research" does not seem exactly trustworthy given that the link to both Codeplex and Google code which have ceased operation quite some time ago. And lastly, one needs to pay for the publication which again sort of raises questions about the trustworthyness, especially given the outdated information and the apparent reliance on authors to host the code elsewhere... (Elsevier also has a "pay to publish" software journal - but they will/offer to host a copy of your code with the paper.)
                          – DetlevCM
                          Aug 29 at 11:27










                        • Their publication fee is quite low in comparison to Elsevier and they originate from a scientific society (the Software Sustainability Institute, so I don't agree with your assessment. (disclaimer: I published with them). You are right with respect to the list of repositories, I will mention it to them.
                          – Pierre de Buyl
                          Aug 29 at 11:42










                        • 325 Pounds vs 500 Euros isn't a huge difference. Now they may have a well reputed backer, but this should be properly communicated AND they should possibly keep their resources up to date. Google Code closed at the beginning of 2016 while Codeplex closed in December 2017. (Doing some more googling now, it seems like Ubiquity Press is normally a "recognised publisher" of open access articles - But the journal presentation does not exactly inspire confidence.)
                          – DetlevCM
                          Aug 29 at 12:51












                        up vote
                        3
                        down vote










                        up vote
                        3
                        down vote









                        The authors provide a link in the article, so the code is either consider considered a reference or part of the research.
                        Whatever the situation, this raises questions:



                        1. Is the code archived? Practical ways to archive code include Zenodo or figshare. Code on a homepage is as good as no code at all.

                        2. Is there a license to the code? If not, its status is not at all evident.

                        As a reviewer, it is up to you to decide what to do. Possible actions include:



                        1. Do not comment on the code.

                        2. Comment on the code with what I would call the minimum: require for the code to be archived and licensed properly.

                        3. Depending on the importance of the computer program in the research, require a minimal amount of readability and that the author provides some tests on the program (i.e. that the program provides known analytical answers if some parameter sets allow it, etc).

                        Regarding archival, you can refer to the editorial information of the Journal of Open Research Software.






                        share|improve this answer












                        The authors provide a link in the article, so the code is either consider considered a reference or part of the research.
                        Whatever the situation, this raises questions:



                        1. Is the code archived? Practical ways to archive code include Zenodo or figshare. Code on a homepage is as good as no code at all.

                        2. Is there a license to the code? If not, its status is not at all evident.

                        As a reviewer, it is up to you to decide what to do. Possible actions include:



                        1. Do not comment on the code.

                        2. Comment on the code with what I would call the minimum: require for the code to be archived and licensed properly.

                        3. Depending on the importance of the computer program in the research, require a minimal amount of readability and that the author provides some tests on the program (i.e. that the program provides known analytical answers if some parameter sets allow it, etc).

                        Regarding archival, you can refer to the editorial information of the Journal of Open Research Software.







                        share|improve this answer












                        share|improve this answer



                        share|improve this answer










                        answered Aug 29 at 11:12









                        Pierre de Buyl

                        99249




                        99249







                        • 1




                          The "Journal of Open Research" does not seem exactly trustworthy given that the link to both Codeplex and Google code which have ceased operation quite some time ago. And lastly, one needs to pay for the publication which again sort of raises questions about the trustworthyness, especially given the outdated information and the apparent reliance on authors to host the code elsewhere... (Elsevier also has a "pay to publish" software journal - but they will/offer to host a copy of your code with the paper.)
                          – DetlevCM
                          Aug 29 at 11:27










                        • Their publication fee is quite low in comparison to Elsevier and they originate from a scientific society (the Software Sustainability Institute, so I don't agree with your assessment. (disclaimer: I published with them). You are right with respect to the list of repositories, I will mention it to them.
                          – Pierre de Buyl
                          Aug 29 at 11:42










                        • 325 Pounds vs 500 Euros isn't a huge difference. Now they may have a well reputed backer, but this should be properly communicated AND they should possibly keep their resources up to date. Google Code closed at the beginning of 2016 while Codeplex closed in December 2017. (Doing some more googling now, it seems like Ubiquity Press is normally a "recognised publisher" of open access articles - But the journal presentation does not exactly inspire confidence.)
                          – DetlevCM
                          Aug 29 at 12:51












                        • 1




                          The "Journal of Open Research" does not seem exactly trustworthy given that the link to both Codeplex and Google code which have ceased operation quite some time ago. And lastly, one needs to pay for the publication which again sort of raises questions about the trustworthyness, especially given the outdated information and the apparent reliance on authors to host the code elsewhere... (Elsevier also has a "pay to publish" software journal - but they will/offer to host a copy of your code with the paper.)
                          – DetlevCM
                          Aug 29 at 11:27










                        • Their publication fee is quite low in comparison to Elsevier and they originate from a scientific society (the Software Sustainability Institute, so I don't agree with your assessment. (disclaimer: I published with them). You are right with respect to the list of repositories, I will mention it to them.
                          – Pierre de Buyl
                          Aug 29 at 11:42










                        • 325 Pounds vs 500 Euros isn't a huge difference. Now they may have a well reputed backer, but this should be properly communicated AND they should possibly keep their resources up to date. Google Code closed at the beginning of 2016 while Codeplex closed in December 2017. (Doing some more googling now, it seems like Ubiquity Press is normally a "recognised publisher" of open access articles - But the journal presentation does not exactly inspire confidence.)
                          – DetlevCM
                          Aug 29 at 12:51







                        1




                        1




                        The "Journal of Open Research" does not seem exactly trustworthy given that the link to both Codeplex and Google code which have ceased operation quite some time ago. And lastly, one needs to pay for the publication which again sort of raises questions about the trustworthyness, especially given the outdated information and the apparent reliance on authors to host the code elsewhere... (Elsevier also has a "pay to publish" software journal - but they will/offer to host a copy of your code with the paper.)
                        – DetlevCM
                        Aug 29 at 11:27




                        The "Journal of Open Research" does not seem exactly trustworthy given that the link to both Codeplex and Google code which have ceased operation quite some time ago. And lastly, one needs to pay for the publication which again sort of raises questions about the trustworthyness, especially given the outdated information and the apparent reliance on authors to host the code elsewhere... (Elsevier also has a "pay to publish" software journal - but they will/offer to host a copy of your code with the paper.)
                        – DetlevCM
                        Aug 29 at 11:27












                        Their publication fee is quite low in comparison to Elsevier and they originate from a scientific society (the Software Sustainability Institute, so I don't agree with your assessment. (disclaimer: I published with them). You are right with respect to the list of repositories, I will mention it to them.
                        – Pierre de Buyl
                        Aug 29 at 11:42




                        Their publication fee is quite low in comparison to Elsevier and they originate from a scientific society (the Software Sustainability Institute, so I don't agree with your assessment. (disclaimer: I published with them). You are right with respect to the list of repositories, I will mention it to them.
                        – Pierre de Buyl
                        Aug 29 at 11:42












                        325 Pounds vs 500 Euros isn't a huge difference. Now they may have a well reputed backer, but this should be properly communicated AND they should possibly keep their resources up to date. Google Code closed at the beginning of 2016 while Codeplex closed in December 2017. (Doing some more googling now, it seems like Ubiquity Press is normally a "recognised publisher" of open access articles - But the journal presentation does not exactly inspire confidence.)
                        – DetlevCM
                        Aug 29 at 12:51




                        325 Pounds vs 500 Euros isn't a huge difference. Now they may have a well reputed backer, but this should be properly communicated AND they should possibly keep their resources up to date. Google Code closed at the beginning of 2016 while Codeplex closed in December 2017. (Doing some more googling now, it seems like Ubiquity Press is normally a "recognised publisher" of open access articles - But the journal presentation does not exactly inspire confidence.)
                        – DetlevCM
                        Aug 29 at 12:51










                        up vote
                        2
                        down vote













                        I'm a software engineer and I want to answer the question from that perspective. Most code is not readable in itself. You need to have comment to document data structures and specifications for subroutine calls. Academics are not software engineers, and I don't expect them to do a professional job in this respect. Still, it is certainly in order to comment on the quality of the software. Without looking at the actual code, I'm not sure I would comment that it is unreadable, because the article (that is according to testimony, sufficient to reproduce the code) is to be considered part of the program documentation. If the program uses short names, that are the same then in the paper, that is no problem. Missing indentation is not an indication of low quality, but many levels of indentation is.
                        I would suggest that you express your feeling that you find it hard to read, but that you are no code expert either, and maybe have some software engineer look at it. It is a different skill set you know. That should take the edge off the comment.



                        To top it off: I've done a good job in cleaning up code, that I didn't understand at the purpose level. You'll be surprised what an expert in a different field is capable of.



                        Bottom line, the code is not of the essence, the quality of the code is incidental and it shouldn't affect your decision either way.






                        share|improve this answer


























                          up vote
                          2
                          down vote













                          I'm a software engineer and I want to answer the question from that perspective. Most code is not readable in itself. You need to have comment to document data structures and specifications for subroutine calls. Academics are not software engineers, and I don't expect them to do a professional job in this respect. Still, it is certainly in order to comment on the quality of the software. Without looking at the actual code, I'm not sure I would comment that it is unreadable, because the article (that is according to testimony, sufficient to reproduce the code) is to be considered part of the program documentation. If the program uses short names, that are the same then in the paper, that is no problem. Missing indentation is not an indication of low quality, but many levels of indentation is.
                          I would suggest that you express your feeling that you find it hard to read, but that you are no code expert either, and maybe have some software engineer look at it. It is a different skill set you know. That should take the edge off the comment.



                          To top it off: I've done a good job in cleaning up code, that I didn't understand at the purpose level. You'll be surprised what an expert in a different field is capable of.



                          Bottom line, the code is not of the essence, the quality of the code is incidental and it shouldn't affect your decision either way.






                          share|improve this answer
























                            up vote
                            2
                            down vote










                            up vote
                            2
                            down vote









                            I'm a software engineer and I want to answer the question from that perspective. Most code is not readable in itself. You need to have comment to document data structures and specifications for subroutine calls. Academics are not software engineers, and I don't expect them to do a professional job in this respect. Still, it is certainly in order to comment on the quality of the software. Without looking at the actual code, I'm not sure I would comment that it is unreadable, because the article (that is according to testimony, sufficient to reproduce the code) is to be considered part of the program documentation. If the program uses short names, that are the same then in the paper, that is no problem. Missing indentation is not an indication of low quality, but many levels of indentation is.
                            I would suggest that you express your feeling that you find it hard to read, but that you are no code expert either, and maybe have some software engineer look at it. It is a different skill set you know. That should take the edge off the comment.



                            To top it off: I've done a good job in cleaning up code, that I didn't understand at the purpose level. You'll be surprised what an expert in a different field is capable of.



                            Bottom line, the code is not of the essence, the quality of the code is incidental and it shouldn't affect your decision either way.






                            share|improve this answer














                            I'm a software engineer and I want to answer the question from that perspective. Most code is not readable in itself. You need to have comment to document data structures and specifications for subroutine calls. Academics are not software engineers, and I don't expect them to do a professional job in this respect. Still, it is certainly in order to comment on the quality of the software. Without looking at the actual code, I'm not sure I would comment that it is unreadable, because the article (that is according to testimony, sufficient to reproduce the code) is to be considered part of the program documentation. If the program uses short names, that are the same then in the paper, that is no problem. Missing indentation is not an indication of low quality, but many levels of indentation is.
                            I would suggest that you express your feeling that you find it hard to read, but that you are no code expert either, and maybe have some software engineer look at it. It is a different skill set you know. That should take the edge off the comment.



                            To top it off: I've done a good job in cleaning up code, that I didn't understand at the purpose level. You'll be surprised what an expert in a different field is capable of.



                            Bottom line, the code is not of the essence, the quality of the code is incidental and it shouldn't affect your decision either way.







                            share|improve this answer














                            share|improve this answer



                            share|improve this answer








                            answered Aug 31 at 13:31


























                            community wiki





                            Albert van der Horst





















                                up vote
                                0
                                down vote













                                My understanding is the authors do not submit the code for publication, so the code does not seem to be subject to your review. The authors just give a link to their code. This begs a question: if they just had provided a reference to their work published elsewhere, would you have considered telling them that that work should be improved? One may argue that validity of their results hinges on the correctness of their code, but they could choose not to provide access to the code at all, as "the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code". You, as a reviewer, took the trouble to verify the results of their computation, thus going beyond and above your duty, but that means that their results are indeed reproducible.



                                So I would think that you may mention that their code is lousy (while apparently correct), but that should not affect your recommendation on publishing / not publishing the paper.






                                share|improve this answer
















                                • 1




                                  If they provided a link to work elsewhere, and following that link lead to a page that was basically unreadable, it wouldn't be a bad idea to point that out to them.
                                  – RDFozz
                                  Aug 31 at 15:13










                                • @RDFozz : That does not seem to contradict the conclusion in the last paragraph of my answer.
                                  – akhmeteli
                                  Aug 31 at 23:29










                                • In my reading of the question, I see nothing that indicates the OP would recommend the paper not be published based on the state of the code (though, perhaps I'm missing something). I was focused on the first couple of sentences of your answer, and trying to point out the difference between a reference to a published work of some sort (which might require actually obtaining a physical copy of the work to look at it, and which is presumably in a format that would not allow simple revision), and a link to a web page (which generally is relatively easy to change/improve).
                                  – RDFozz
                                  Sep 1 at 0:42










                                • @RDFozz : There may be some practical difference, but still the code is beyond the scope of review, and the authors may have all kinds of reasons not to touch the code.
                                  – akhmeteli
                                  Sep 1 at 1:27














                                up vote
                                0
                                down vote













                                My understanding is the authors do not submit the code for publication, so the code does not seem to be subject to your review. The authors just give a link to their code. This begs a question: if they just had provided a reference to their work published elsewhere, would you have considered telling them that that work should be improved? One may argue that validity of their results hinges on the correctness of their code, but they could choose not to provide access to the code at all, as "the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code". You, as a reviewer, took the trouble to verify the results of their computation, thus going beyond and above your duty, but that means that their results are indeed reproducible.



                                So I would think that you may mention that their code is lousy (while apparently correct), but that should not affect your recommendation on publishing / not publishing the paper.






                                share|improve this answer
















                                • 1




                                  If they provided a link to work elsewhere, and following that link lead to a page that was basically unreadable, it wouldn't be a bad idea to point that out to them.
                                  – RDFozz
                                  Aug 31 at 15:13










                                • @RDFozz : That does not seem to contradict the conclusion in the last paragraph of my answer.
                                  – akhmeteli
                                  Aug 31 at 23:29










                                • In my reading of the question, I see nothing that indicates the OP would recommend the paper not be published based on the state of the code (though, perhaps I'm missing something). I was focused on the first couple of sentences of your answer, and trying to point out the difference between a reference to a published work of some sort (which might require actually obtaining a physical copy of the work to look at it, and which is presumably in a format that would not allow simple revision), and a link to a web page (which generally is relatively easy to change/improve).
                                  – RDFozz
                                  Sep 1 at 0:42










                                • @RDFozz : There may be some practical difference, but still the code is beyond the scope of review, and the authors may have all kinds of reasons not to touch the code.
                                  – akhmeteli
                                  Sep 1 at 1:27












                                up vote
                                0
                                down vote










                                up vote
                                0
                                down vote









                                My understanding is the authors do not submit the code for publication, so the code does not seem to be subject to your review. The authors just give a link to their code. This begs a question: if they just had provided a reference to their work published elsewhere, would you have considered telling them that that work should be improved? One may argue that validity of their results hinges on the correctness of their code, but they could choose not to provide access to the code at all, as "the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code". You, as a reviewer, took the trouble to verify the results of their computation, thus going beyond and above your duty, but that means that their results are indeed reproducible.



                                So I would think that you may mention that their code is lousy (while apparently correct), but that should not affect your recommendation on publishing / not publishing the paper.






                                share|improve this answer












                                My understanding is the authors do not submit the code for publication, so the code does not seem to be subject to your review. The authors just give a link to their code. This begs a question: if they just had provided a reference to their work published elsewhere, would you have considered telling them that that work should be improved? One may argue that validity of their results hinges on the correctness of their code, but they could choose not to provide access to the code at all, as "the mathematics underlying these computations is explained sufficiently well that it is possible to reproduce the results without using the authors' code". You, as a reviewer, took the trouble to verify the results of their computation, thus going beyond and above your duty, but that means that their results are indeed reproducible.



                                So I would think that you may mention that their code is lousy (while apparently correct), but that should not affect your recommendation on publishing / not publishing the paper.







                                share|improve this answer












                                share|improve this answer



                                share|improve this answer










                                answered Aug 31 at 6:17









                                akhmeteli

                                66329




                                66329







                                • 1




                                  If they provided a link to work elsewhere, and following that link lead to a page that was basically unreadable, it wouldn't be a bad idea to point that out to them.
                                  – RDFozz
                                  Aug 31 at 15:13










                                • @RDFozz : That does not seem to contradict the conclusion in the last paragraph of my answer.
                                  – akhmeteli
                                  Aug 31 at 23:29










                                • In my reading of the question, I see nothing that indicates the OP would recommend the paper not be published based on the state of the code (though, perhaps I'm missing something). I was focused on the first couple of sentences of your answer, and trying to point out the difference between a reference to a published work of some sort (which might require actually obtaining a physical copy of the work to look at it, and which is presumably in a format that would not allow simple revision), and a link to a web page (which generally is relatively easy to change/improve).
                                  – RDFozz
                                  Sep 1 at 0:42










                                • @RDFozz : There may be some practical difference, but still the code is beyond the scope of review, and the authors may have all kinds of reasons not to touch the code.
                                  – akhmeteli
                                  Sep 1 at 1:27












                                • 1




                                  If they provided a link to work elsewhere, and following that link lead to a page that was basically unreadable, it wouldn't be a bad idea to point that out to them.
                                  – RDFozz
                                  Aug 31 at 15:13










                                • @RDFozz : That does not seem to contradict the conclusion in the last paragraph of my answer.
                                  – akhmeteli
                                  Aug 31 at 23:29










                                • In my reading of the question, I see nothing that indicates the OP would recommend the paper not be published based on the state of the code (though, perhaps I'm missing something). I was focused on the first couple of sentences of your answer, and trying to point out the difference between a reference to a published work of some sort (which might require actually obtaining a physical copy of the work to look at it, and which is presumably in a format that would not allow simple revision), and a link to a web page (which generally is relatively easy to change/improve).
                                  – RDFozz
                                  Sep 1 at 0:42










                                • @RDFozz : There may be some practical difference, but still the code is beyond the scope of review, and the authors may have all kinds of reasons not to touch the code.
                                  – akhmeteli
                                  Sep 1 at 1:27







                                1




                                1




                                If they provided a link to work elsewhere, and following that link lead to a page that was basically unreadable, it wouldn't be a bad idea to point that out to them.
                                – RDFozz
                                Aug 31 at 15:13




                                If they provided a link to work elsewhere, and following that link lead to a page that was basically unreadable, it wouldn't be a bad idea to point that out to them.
                                – RDFozz
                                Aug 31 at 15:13












                                @RDFozz : That does not seem to contradict the conclusion in the last paragraph of my answer.
                                – akhmeteli
                                Aug 31 at 23:29




                                @RDFozz : That does not seem to contradict the conclusion in the last paragraph of my answer.
                                – akhmeteli
                                Aug 31 at 23:29












                                In my reading of the question, I see nothing that indicates the OP would recommend the paper not be published based on the state of the code (though, perhaps I'm missing something). I was focused on the first couple of sentences of your answer, and trying to point out the difference between a reference to a published work of some sort (which might require actually obtaining a physical copy of the work to look at it, and which is presumably in a format that would not allow simple revision), and a link to a web page (which generally is relatively easy to change/improve).
                                – RDFozz
                                Sep 1 at 0:42




                                In my reading of the question, I see nothing that indicates the OP would recommend the paper not be published based on the state of the code (though, perhaps I'm missing something). I was focused on the first couple of sentences of your answer, and trying to point out the difference between a reference to a published work of some sort (which might require actually obtaining a physical copy of the work to look at it, and which is presumably in a format that would not allow simple revision), and a link to a web page (which generally is relatively easy to change/improve).
                                – RDFozz
                                Sep 1 at 0:42












                                @RDFozz : There may be some practical difference, but still the code is beyond the scope of review, and the authors may have all kinds of reasons not to touch the code.
                                – akhmeteli
                                Sep 1 at 1:27




                                @RDFozz : There may be some practical difference, but still the code is beyond the scope of review, and the authors may have all kinds of reasons not to touch the code.
                                – akhmeteli
                                Sep 1 at 1:27





                                protected by Alexandros Aug 31 at 15:39



                                Thank you for your interest in this question.
                                Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).



                                Would you like to answer one of these unanswered questions instead?


                                Popular posts from this blog

                                How to check contact read email or not when send email to Individual?

                                Displaying single band from multi-band raster using QGIS

                                How many registers does an x86_64 CPU actually have?