gray

Applied Program Evaluation Research: Neighborhoods Now Working Group

Neighborhoods Now is an initiative spearheaded by the Van Alen Institute in collaboration with Urban Design Forum. This project began in 2020, connecting neighborhoods in New York City that had been especially affected by the COVID-19 pandemic with design firms in the Van Alen/Urban Design Forum network offering pro bono design work.

This hypothetical program evaluation combines multi-modal research to examine the success of one partnership within the program (Community League of the Heights [CLOTH] and Arup) with the goal of providing research-based evidence of the importance of these types of partnerships, and to ultimately encourage more work between designers and communities in the future. 

Collaborators: Advised by Dr. Kumbie Madondo
Client: Completed as hypothetical exercise
Industry: Urban Design, Urban Planning
Skills: Program Evaluation, Qualitative Research, Quantitative Research, Research Design
Timeline: Completed January 2021-May 2021
Tools: Google Suite

Note: This case study was not completed in affiliation with Neighborhoods Now, Van Alen Institute, or Urban Design Forum. 

DOWNLOAD FULL PAPER HERE

INTRODUCTION

Research Question

Has the collaboration between designers (Arup) and community-based organization (CLOTH) led to safe and community-focused recovery/reopening of businesses? Additionally, what was the overall impact on individual community members?

My Role

This project was a hypothetical exercise developed under Dr. Kumbie Madondo as part of a requirement for a graduate-level course on applied evaluation research. To come up a  with a robust answer to the research question,  I developed a multi-pronged approach involving both outcome and process based evaluation components, utilizing quantitative and qualitative research methods. 

LITERATURE REVIEW

The literature review revealed two main takeaways: (1) access to equitable public spaces became even more important during COVID-19, and (2) for a public space to be equitable, communities need to be allowed to provide input on their design and programming

Importance of Equitable Public Space

1. Design interventions that increase the adaptive capacity of public spaces during the COVID-19 pandemic can also help make public spaces more equitable.¹

2. Exclusion from public space, "as women were from the Athenian Agora or African Americans were (and too often still are) in segregated America, is one of the clearest markers of being less than a full member of the civitas,”.²

3. Access to quality public spaces during COVID-19 has allowed some continuation of "normal" life. 

Jump to references.

Research suggests that design interventions to public space during the COVID-19 pandemic may be an important way to establish equity among communities in New York City.

Community Participation in Public Space Design

1. Citizen participation in planning is vital to equitable public space interventions.³

2. A trusted community organization can be a crucial ally for pandemic relief interventions that aid individual community members.

3. Extensive stakeholder interest can lead to a lack of trust⁴, however, so it's important to work with organizations that are trusted by their communities. 

Jump to references.

STAKEHOLDERS

Considered Organizations & Groups

pexels-laura-tancredi-7078479-copy

Community League of the Heights (CLOTH)

CLOTH is a multi-service community development organization dedicated to supporting and empowering economically disadvantaged residents of Washington Heights, and whose experiences in the Neighborhoods Now program may determine how they collaborate with design agencies in the future.

battery-park-nyc-untapped-cities1-1-1-copy-2

Arup

Arup is a global firm of designers, planners, engineers, architects, consultants, and technical specialists, and this evaluation may help the firm to better understand the best ways to collaborate with community organizations.

mauro-mora-31-pOduwZGE-unsplash-copy

Van Alen Institute

Van Alen Institute employs community-centered design to help create more equitable and just cities. Their work allows for input from diverse voices in the creation of the built environment. Van Alen Institute helped connect community organizations with design, architecture, and other affiliations within the Neighborhoods Now project.

pexels-pixabay-162024-copy-2

Urban Design Forum

Urban Design Forum works to interrogate the challenges of the built environment by convening civic leaders from various backgrounds in New York City. They utilized their network of partnerships for the Neighborhoods Now project.

oskars-sylwan-99wmYtakIyg-unsplash-copy-2

Other Supporting Firms

Other supporting firms in this partnership included Design Advocates, Gensler, School of Jellyfish, Stantec, and Woods Bagot.

conor-brown-q32NPl_biRU-unsplash

Small Businesses

Small businesses in Washington Heights would benefit from outdoor structures, whether that is outdoor seating or outdoor pop-up shops.

dylan-gillis-KdeqA3aTnBY-unsplash

Community Members

Individual community members would benefit from being able to gather in outdoor public spaces and continue patronizing small businesses.

 

 

Stakeholder Influence

RFP-Presentation-2

OVERVIEW

The Washington Heights Working Group

This working group addressed several components of pandemic recovery, including: return to school logistics, Open Restaurant seating, utilization of NYC Open Streets, distribution of informational materials on creating pop-up shops, and small business support through outdoor structure design. This evaluation will assess Arup and CLOTH’s collaboration on physical, outdoor interventions only, which include the Open Restaurant seating, utilization of NYC Open Streets, and small business support through outdoor structure design components of the program.

Components to Evaluate:

Circle-2

1. Open Restaurant Seating

Design partner Woods Bagot implemented a pilot design of a year-round seating structure at local business Colombian Coffee & Kitchen. The goal of this structure is to be implemented at other local businesses after this trial run.

2. Playstreets

Stantec and CLOTH applied for five playstreets through the “NYC Open Streets” program. They also created a design proposal for Plaza de las Americas involving pop-up gardens and a collective restaurant seating area.

Thanks to NYC DOT for this image, which has been cropped with a duotone filter applied.

Circle-1
Circle-3

3. Outdoor Kit-of-Parts

The team designed a flexible outdoor kit-of-parts in collaboration with Word Up Community Bookshop. The kit-of-parts could be used for outdoor educational programming, civic gathering space, or help local businesses move their operations outdoors.

Urban design interventions can help optimize adaptive capacity of public spaces.

EVALUATION QUESTIONS

The following questions that this evaluation aims to answer are designated as process versus outcome evaluation questions, and are further categorized as having an organizational level or community level focus.

Process Evaluation

Organizational Level Questions

1. What criteria determines which businesses are able to benefit most from outdoor design interventions?
2. Do businesses have the resources to maintain the design interventions?

Community Level Questions

3. How did the outdoor design interventions allow community members to utilize public spaces during the coronavirus pandemic?
4. Which community members were able to access the design interventions?

Outcome Evaluation

Organizational Level Questions

1. How did the design resources made available by the Van Alen Institute and the Urban Design Forum combined with the local expertise of CLOTH alleviate the needs of the Washington Heights businesses hit hardest by the pandemic?
2. How can the partnership between Arup and CLOTH continue beyond pandemic relief?

Community Level Questions

3. Did the partnership between Arup and CLOTH increase overall community access to usable public space during the pandemic?
4. How have the design interventions changed how community members gather in public?

METHODS

Organizational Level Data Collection

To evaluate the program at the organizational level, semi-structured interviews will be conducted with CLOTH, Arup and design partner organization leaders as well as small business owners within the community. Below, evaluation questions are listed for every permutation of the evaluation.

NN_research-matrix_1-01

What criteria determines which businesses are able to benefit most from outdoor design interventions (such as outdoor restaurant seating or the outdoor kit-of-parts)?

This question will be answered by CLOTH, Arup, and design partners. Interviews among leaders involved from the outset of the program will be most crucial in order to gain the perspective of those involved with choosing the types of outdoor seating related interventions necessary to alleviate some of the strains of the pandemic on small businesses.

Do businesses have the resources to maintain the design interventions?

Interview participants will consist of business owners of businesses that participated in the Open Restaurant Seating and Open Streets initiatives spearheaded by Neighborhoods Now. It will be important to understand both what these owners know about caring for the outdoor structures they use in addition to how to address possible changes in zoning or policy regarding outdoor structures.

NN_research-matrix_3-01

How can the partnership between Arup and CLOTH continue beyond pandemic relief?

Answering this question would require interviewing those involved at the highest level, as these participants would be most likely to provide an accurate picture of the feasibility of continuing the program beyond the pandemic. The semi-structured interviews would require an interview protocol that may be deviated from during the conversation should the interviewee provide important feedback on a topic that may not have been initially thought to be relevant.

How did the design resources made available by the Van Alen Institute and the Urban Design Forum combined with the local expertise of CLOTH alleviate the needs of the Washington Heights businesses hit hardest by the pandemic?

This question will require interviewing business owners who utilized the program services provided by Neighborhoods Now and whose businesses operated at least 2-3 years prior to the onset of the pandemic, such as Colombian Coffee & Kitchen. This will allow for responses that create a fair comparison between how the needs of the businesses changed due to the pandemic, and how those needs were alleviated by Neighborhoods Now.

Community Level Data Collection

Community-level data collection will require a more nuanced approach due to the nature of recruiting community participants. This approach will also be divided between process and outcome based questions, unlike organizational level data collection, which was divided by type of interview subject. Process-based strategies will involve surveys, followed up by interviews based on survey responses, where outcome-based strategies will utilize focus groups for data collection.

NN_research-matrix_2-01

1. How did the outdoor design interventions allow community members to utilize public spaces during the coronavirus pandemic?
2. Which community members were able to access the design interventions?

In order to collect data for process evaluation questions at the community level, participants will be recruited for surveys through the small businesses they patronize that received services from the Neighborhoods Now program. Advertisements for the survey may be distributed via community newsletter, bulletin board in public areas, or by working with CLOTH to advertise for the survey directly through materials for the small businesses that received services from Neighborhoods Now (such as posted on a bulletin board in Colombian Coffee & Kitchen).

NN_research-matrix_4-01

1. Did the partnership between Arup and CLOTH increase overall community access to usable public space during the pandemic?
2. How have the design interventions changed how community members gather in public?

Outcome evaluation questions will be addressed via focus groups with selected community members. Focus group members will be recruited in many of the same ways as process-level survey and interview respondents, including via community advertisements or through following up with survey participants (again, permission to do so will be requested in the survey).

Sample Survey Questions

Prior to the pandemic, how often did you visit Plaza de las Americas?
Frequently
Occasionally
Rarely
Never
Other: __________________________

During the pandemic, AFTER [design intervention], how often did you utilize Plaza de las Americas?
Frequently
Occasionally
Rarely
Never
Other: __________________________

During the pandemic, how often did you dine outdoors?
Frequently
Occasionally
Rarely
Never
Other: __________________________

If you answered A. or B. to the previous question, where were/are your favorite places to dine outdoors? ____________________________________________________________

How often do you pass a public playstreet (also known as a street that is closed for pedestrian use)?
Frequently
Occasionally
Rarely
Never
Other: __________________________

To what extent do you enjoy spending time at Plaza de las Americas?
To a great extent
Somewhat
Very little
Not at all
No opinion - I do not spend time there
Other: __________________________

If you answered A. or B. to the previous question, which activities do you participate in at Plaza de las Americas?
Hanging out with friends
Taking an outdoor class
Eating a meal
Celebration (Birthday, graduation, or other)
I take my children to play there
Other

Sample Follow-up Interview Protocol Questions

Can you tell me about a notable experience (can be positive or negative) you have had in Plaza de las Americas?
What about a notable experience (again, positive or negative) after the outdoor structures were introduced to Plaza de las Americas?
How easy is it for you to get to a public playstreet? How do you feel when you’re there?
Do you ever see anyone you know there? If so, can you tell me about them?

DATA ANALYSIS

Qualitative Analysis Plan

Coding for this evaluation would be conducted using RQDA, a free CAQDAS (computer-assisted qualitative data analysis software) package for R. Grounded theory methodologies would be employed during the coding process.

First cycle coding will utilize concept coding, in which the qualitative data collected will be characterized by concepts related to their respective research questions. A provisional list of concept codes will be determined prior to coding the data so that multiple researchers can code the interviews and maintain the same evaluative framework. Second cycle coding will utilize axial coding techniques for determining larger categories within the smaller concepts found during first cycle coding. Axial coding also lends itself to visualization via “mind map” types of diagrams, which will be useful for data visualization and dissemination.

NN_Sketch-maps-01

Quantitative Analysis Plan

NN_Sketch-maps-02

Community-level data collected via surveys will be analyzed in Stata using analysis of variance (ANOVA) and multiple regression analysis. The ANOVA will be conducted to help compare the experiences of different community groups both before the design intervention was implemented and afterwards.

Multiple regression analysis will be utilized to help further understand any differences in the experience of the design intervention by different community groups.

For example, while the ANOVA can help determine whether different demographics were affected differently by the implementation of design interventions, multiple regression analysis can help determine if the relationship between the “success” (or “failure”) of the design intervention can be attributed to other factors beyond demographics alone.

Data Dissemination

Findings will be shared with major stakeholders (Van Alen Institute, CLOTH, Arup) via presentation with data visualization graphics to help explain some of the findings of the evaluation.

The presentation itself will be shared with the major stakeholders to share with any of their partners who participated in the Neighborhoods Now program. The presentation will be simplified into a pamphlet that can be shared with other stakeholders who may be interested in the results of the evaluation, such as community members or small businesses.

CONCLUSION

Conclusion

This evaluation aims to assess the specific collaboration between Arup and CLOTH in Washington Heights, and whether their efforts in aiding outdoor gatherings via design structures have allowed for community and small business focused recovery during the coronavirus pandemic. pandemic. In order to accomplish this, the evaluation calls for both qualitative and quantitative methods in collecting data to answer both process and outcome based evaluation questions.

The success of this evaluation in generating analyses that can be applied to other aspects of the Neighborhoods Now program would imply that the research produced here could help influence pro-bono design work for community organizations in the future.

It can also support previous research⁴ that has underlined the importance of the role of trusted organizations and institutions in facilitating the general public’s input in urban planning. Ultimately, the evaluation will provide insight for the Van Alen Institute/Urban Design Forum for the most useful ways to continue leveraging their partnerships for public good, and set precedents for ways that future urban design organizations can do the same.

DOWNLOAD FULL PAPER HERE

References

1. Stevens, Nicholas J., Silvia G. Tavares, and Paul M. Salmon. “The Adaptive Capacity of Public Space under COVID‐19: Exploring Urban Design Interventions through a Sociotechnical Systems Approach.” Human Factors and Ergonomics in Manufacturing & Service Industries 31, no. 4 (July 2021): 333–48. https://doi.org/10.1002/hfm.20906.

2. Kasinitz, Philip. “Rending the ‘Cosmopolitan Canopy’: COVID–19 and Urban Public Space.” City & Community 19, no. 3 (September 2020): 489–95. https://doi.org/10.1111/cico.12516.

3. Eriksson, Erik, Amira Fredriksson, and Josefina Syssner. “Opening the Black Box of Participatory Planning: A Study of How Planners Handle Citizens’ Input.” European Planning Studies, March 4, 2021, 1–19. https://doi.org/10.1080/09654313.2021.1895974.

4. Menzel, Susanne, Matthias Buchecker, and Tobias Schulz. “Forming Social Capital—Does Participatory Planning Foster Trust in Institutions?” Journal of Environmental Management 131 (December 2013): 351–62. https://doi.org/10.1016/j.jenvman.2013.10.010.