top of page

Hidden Research Biases in Modern Methods

Updated: Jan 6


While reading "Softwar" by Major Jason P Lowry of the United States Space Force, I noticed an interesting choice in his research approach. He used grounded theory to study how power dynamics work in nature and warfare. Instead of starting with a set hypothesis, this method allows patterns and theories to emerge naturally from the data collected.


The "Where There's Smoke, There's Fire" Analogy


The "smoke and fire" analogy is an excellent way to understand grounded theory's approach. Just as seeing smoke leads us to investigate the presence of fire, grounded theory researchers:


  • Observe initial phenomena (the "smoke")

  • Systematically gather data about everything surrounding that observation

  • Follow the evidence trail to discover the underlying causes (the "fire")

  • Build theories based on what the collected evidence reveals


This approach differs from traditional research methods because instead of starting with a theory about what's causing the smoke and trying to prove it, researchers carefully document and analyze all aspects of the smoke - its patterns, density, color, movement - and let these observations guide them to understanding the nature of the fire.


The key principle is that comprehensive data collection and analysis around an observed phenomenon (the smoke) will naturally lead to understanding its root causes and relationships (the fire), resulting in theories that are firmly grounded in real-world evidence.


Why Use Grounded Theory?


Grounded theory is particularly valuable when researchers need to understand complex patterns and relationships that emerge from data. This methodology allows researchers to discover insights that might be missed by traditional hypothesis-driven approaches, making it especially useful when studying intricate social processes or organizational dynamics where predetermined theories might limit understanding.


Other Research Methods and Limitations


Every research method has its weaknesses. Researchers can introduce bias, misapply methods, or run into limitations that affect both academic studies and real-world applications. While formal research often requires advanced academic training, many professionals naturally use similar approaches in their daily work - even if they don't call it "research." These practical applications can be just as valuable as formal academic studies.


Eleven Core Methodologies and Their Inherent Flaws


Understanding research methodologies is crucial for making informed and reliable decisions. Different approaches provide unique frameworks for evaluating options and reaching conclusions, though each comes with its own potential biases that decision-makers must carefully consider. From confirmation bias in hypothesis formation to the challenges of maintaining objectivity, recognizing these limitations helps ensure more rigorous and trustworthy research outcomes.


1. Scientific Method

  • Confirmation bias distorts hypothesis formation

  • Observer effects alter experimental results

  • Excessive focus on measurable aspects while neglecting unmeasurable ones

  • Publication bias favors positive results


2. Pragmatism

  • Overemphasis on practical outcomes over theoretical understanding

  • Risk of methodological inconsistency when mixing approaches

  • Difficulty maintaining research rigor while prioritizing utility

  • Challenge of balancing immediate needs with long-term research value


3. Case Study Research

  • Bias in case selection

  • Overemphasis on unique features

  • Difficulty proving cause and effect

  • Limited ability to generalize findings


4. Action Research

  • Conflicts between researcher and participant roles

  • Emotional involvement compromises objectivity

  • Pressure to show positive outcomes

  • Challenges in maintaining research rigor


5. Systems Theory

  • Complex relationships oversimplified

  • Critical variables overlooked

  • System boundaries poorly defined

  • Structure emphasized over process


6. Grounded Theory

  • Personal biases influence data interpretation

  • Theories formed before sufficient data collection

  • Data forced into predetermined categories

  • Inconsistent coding practices


7. Critical Theory

  • Ideological bias affects analysis

  • Activist goals compromise objectivity

  • Power dynamics between researchers and subjects

  • Evidence interpreted selectively


8. Complexity Theory

  • Over-attribution of emergent properties

  • True causality difficult to identify

  • Complex patterns oversimplified

  • Models incorrectly specified


9. Ethnographic Research

  • Researcher's cultural background creates bias

  • Researcher's presence alters natural behavior

  • Selective observation of phenomena

  • Over-identification with research subjects


10. Phenomenological Research

  • Interview bias affects data collection

  • Participants' memories prove unreliable

  • Researchers struggle to set aside preconceptions

  • Individual experiences over-generalized


11. Mixed Methods

  • Poor integration of methodological approaches

  • Difficulty resolving contradictory findings

  • Methodological incompatibilities

  • Increased complexity breeds errors


Implications for Research Methods


Each research method has its own limitations. Researchers need to be humble and recognize that one method alone may only show part of the picture. The best approach is often to combine different methods thoughtfully to get a more complete understanding..


Real-World Example: Railway Projects


Let's look at how different research methods work together in railway projects:


  • Systems Theory in Practice: Teams analyze interconnected components and their relationships, though they must avoid oversimplifying complex systems and overlooking critical variables


  • Grounded Theory Application: Teams collect and analyze historical project data to identify patterns, being careful not to form conclusions before gathering sufficient evidence


  • Case Study Research: Teams examine previous projects while acknowledging the limitations in generalizing findings to new situations


  • Action Research Methods: Teams implement and test solutions iteratively, maintaining research rigor while adjusting based on practical outcomes


For instance, when deciding whether to close a station for repairs, teams might use multiple research approaches:


  • Systems Theory: Create different scenario models to understand system-wide impacts


  • Grounded Theory: Analyze emerging patterns from collected data to refine the approach


  • Case Study Research: Study similar station repair projects and their outcomes


  • Action Research: Talk to stakeholders and gather direct feedback


This practical application demonstrates how research methods work together to solve real-world problems.


Protecting Research Integrity


Financial interests and funding sources can profoundly shape research outcomes through subtle yet significant ways, from influencing study design and methodology to affecting which results get emphasized or published.


When financial stakeholders have vested interests in specific outcomes, there's a risk of introducing bias at various stages of the research process. To maintain scientific integrity, it's crucial to carefully examine how studies are funded, designed, and conducted, while actively questioning methodologies and potential conflicts of interest:


  • Scientific Studies: Funding can affect what gets studied and published

  • Theory Building: Financial interests might skew data interpretation

  • Case Selection: Funders might cherry-pick favorable examples

  • Complex Studies: Large projects may be influenced by sponsors

  • Critical Research: Some topics might be avoided if they threaten powerful interests


To protect research integrity, we should:

  • Be open about funding sources

  • Have others repeat the research

  • Use independent reviewers

  • Share data openly


While we can't eliminate all bias, we can work to minimize it.


Conclusion


Every research method has strengths and weaknesses. While some methods work better for certain tasks (like studying power dynamics), no single approach is perfect. Modern tech and society are complex - we need to combine methods carefully, stay aware of bias, and remain open to new approaches.


Appendix: Research Methods Overview and Common Flaws

Method

Description

Time Period & Origin

Best For

Not Suitable For

Common Human Errors/ Flaws

1. Scientific Method

Hypothesis-driven, experimental approach

17th century, Francis Bacon's empiricism

Controlled experiments, replicable studies

Complex social phenomena

Confirmation bias in hypothesis formation, observer effect influencing results, overemphasis on measurable aspects

2. Pragmatism

Practical approach focusing on what works

Late 1800s, Charles Sanders Peirce & William James

Real-world problem solving, practical applications

Pure theoretical research

Over-emphasis on immediate utility, neglect of theoretical foundations, oversimplification of complex issues

3. Case Study Research

In-depth analysis of specific instances

1920s, Chicago School of Sociology

Detailed contextual analysis

Broad generalizations

Selection bias in choosing cases, over-emphasis on unique aspects, limited generalizability

4. Action Research

Practical problem-solving approach

1940s, Kurt Lewin at MIT

Organizational change, practical improvements

Pure theoretical research

Dual-role conflict, emotional involvement affecting objectivity, pressure for positive outcomes

5. Systems Theory

Study of interrelated components

1950s, Ludwig von Bertalanffy

Complex systems analysis

Simple linear relationships

Oversimplification of complex relationships, missing crucial variables, difficulty in boundary definition

6. Grounded Theory

Theory emerges from systematic data analysis

1960s, Glaser & Strauss

Complex social processes, theory building

Testing existing theories

Personal biases affecting interpretation, premature theory formation, forcing data into preconceived categories

7. Critical Theory

Analysis of power and social structures

1930s, Frankfurt School

Social critique, power analysis

Value-neutral research

Ideological bias, activist agenda affecting objectivity, selective interpretation of evidence

8. Complexity Theory

Study of complex adaptive systems

1960s, Santa Fe Institute

Complex systems, emergent behavior

Simple linear relationships

Over-attribution to emergence, difficulty in identifying true causality, oversimplification of patterns

9. Ethnographic Research

In-depth study of cultures and groups

Early 1900s, Bronisław Malinowski

Cultural understanding, social dynamics

Large-scale quantitative analysis

Cultural bias from researcher's background, presence affecting natural behavior, selective observation

10. Phenomenolo-gical Research

Study of lived experiences

Early 1900s, Edmund Husserl

Understanding personal experiences

Large population studies

Interview bias, memory distortion in participants, difficulty in bracketing researcher's preconceptions

11. Mixed Methods

Combination of approaches

1980s-1990s, Pragmatic Movement

Multi-faceted phenomena

Pure theoretical concepts

Poor integration of methods, contradictory findings handling, increased complexity leading to errors


 
 
bottom of page