Peer Reviewing Papers

Mon, 01 Jan 0001 00:00:00 +0000

[in development]

Selecting Scientific Tools

Mon, 01 Jan 0001 00:00:00 +0000

This page started as a lab meeting discussion of how to choose among different options for scientific tools (mostly focused on programming).

Question Framing

When should I learn python/R/Stan/etc.?
How does {tool x} integrate into my workflow?
If I don’t learn {tool x}, what am I doing instead?
What are the benefits for knowing how to use {tool x}
What are the benefits for learning how to use {tool x}

Identify Alternatives

some possible scenarios
- I have a specific research problem that doesn’t appear to solvable with my current toolset
- Lots of people are using {tool x}, and I don’t want to be left behind
- No one else is using {tool x}, and I want to be the leading expert in my lab/department/field
what are some other considerations beyond solving my specific problem
- speed
- generality
how do I find out about other options?
- google +3
- twitter +3
- literature search
- seminar talks, conferences (people tend to be a little more “in the weeds”, more receptive for engagement & dialogue)
- workshops
what are the people around me using?
- peers, colleagues, collaborators
- similar enough level for learning/teaching

Evaluate the utility of knowing a new tool

Ok, @cmMcConnaughy is right - enough language war. My language-agnostic concluding thoughts. Choose tools that:
1) Have strong communities around them
2) Are proven but adaptive, so you can use them long-term
3) Facilitate persistent reproducibility, esp. w/ your future self
— Thomas J. Leeper (@thosjleeper) February 4, 2019

properties of communities to consider:

communities to avoid (non-inclusive, wrong level)
asking about what communities are set up around tools
how diverse is the steering committee for the tool
what are the practices of the community?
If you are going to spend time in a community, will you be able to grow and take on leadership roles? (is that important for you)

What is the long-term trajectory of a tool:

is it actively being developed?
established history of developers
support for older versions

Evaluate the utility of learning a new tool

humans are habit-forming
- defer learning new tools –> (decades later) “I’m too old to learn {tool x}”
- always learning new tools –> never get around to using anything in practice
exposure to new communities builds your professional network
exposure to new ideas gives added context to what you already knew before
“learning tools” is also a skill! (which means you can practice it and get better)
“if you don’t use it, you lose it” -

Things to keep in mind

Decision making is difficult (but it is a skill you can practice and improve on!)

imperfect knowledge
- no information source is incomplete
- information sources aren’t completely tuned to your individual situation
outcomes are imperfectly measured
- how useful was learning {tool x}? (no ideal way to measure this)
- no measurement of what would have occurred with an alternative
limited time/attention
- spending (more) effort to improve decisions can also be suboptimal
when to abandon bad decisions (sunk cost fallacy)

Tips

find good heuristics
- be aware of biases (cognitive and otherwise)
- reflect on past decisions (successful and unsuccessful)
engage in deliberate practice
- identify appropriate goals
- evaluate outcomes
- get targeted feedback
- repeat
embrace failure
- “People don’t fail, processes do”
- debug your processes (learn from mistakes, aim for long-term growth)
learn to fail fast
- for large/complex projects - is {tool x} suitable long-term?
- test whether the {tool x} addresses the most difficult aspect of the project
  - if this doesn’t work, it is better to know sooner!
exposure vs. learning
- goal-oriented or not?

Professional Development | Weecology Wiki