Cómo citar:

How to cite:

Aliaga, Francisco M. (2014). Twenty years of electronic publishing and open access: a pioneer reaches maturity.  RELIEVE, v. 20 (1), art. 0. DOI: 10.7203/relieve.20.1.3856


This article in Spanish (Original Version)


  Hits:  Hit counter - Contador de visitas Visitas  


[Veinte años de publicación electrónica y de acceso abierto: la madurez de una pionera]


  In article in pdf format

por / by

En formato artículo en pdf



About the authors

Print this article

Aliaga, Francisco M. (Francisco.Aliaga@uv.es)

Ficha del artículo

 Sobre los autores

Imprimir el artículo




    RELIEVE reaches 20 years, a milestone in electronic publishing and the open access to educational research. We review some of the achievements and future prospects, as well as analyze some circumstances of journal evaluation processes.



   RELIEVE alcanza los 20 años, lo que supone un hito en la edición electrónica y en el acceso abierto a la investigación educativa. Revisamos algunos de los logros conseguidos y las perspectivas de futuro, a la vez que analizamos algunas circunstancias de los procesos de evaluación de revistas.  


 Journal, journal articles, open access, Publishing, electronic journals, evaluation, electronic publishing.



   Revista,, acceso abierto, edición, revistas electrónicas, artículos de revista, evaluación, edición electrónica. 


Comentar este artículo

Artículos relacionados:

Aliaga & Suárez-Rodríguez(2008). La repercusión científica de una revista académica: análisis del caso de RELIEVE 

Volumen 20, n. 1

     On July 20, 1994 the National Library of Spain awarded the ISSN number to a new magazine called RELIEVE – The Journal of Educational Research and Evaluation (Rodríguez Gómez, 2005). It was not just a new scientific journal, it also marked a new period in academic communication, since it was the first Spanish electronic journal (and quite possibly the first electronic journal written in Spanish).

    In that year, Roes (1994) identified just 34 electronic journals in the entire world, but many of them were only accessible through restricted networks (through intranets). However, a new method of scientific communication was emerging that would quickly come to change academic habits and customs.  

Among the main advantages of electronic journals, according to Rodríguez-Gómez (1999) are the following:

  • Electronic journals are cheaper to produce.

  • They are quicker to distribute.

  • They have a broader reach.

  •  They are more accurate and well-written.

  • They are easier to read.

  • They are easier to store.

  • They are easier to control.

 Electronic Journals

     Electronic journals have had (and still have) to overcome many prejudices. Very often, in the higher levels of public administration as well as in the academic world, conservatism and resistance to innovation have led to undervaluing of electronic journals. This is the case, for instance, in the context of research work done by university professors in Spain. The regulations have for many years (and even now, in some specialist areas) applied radically different (and more restrictive) quality criteria to electronic journals compared to print – as if the true determining factor were the format rather than the content.

     Electronic journals provide a wide range of new possibilities, as Aliaga and Suárez-Rodríguez (2002) show. Unfortunately, traditionalist inertia, technological difficulties or lack of interest mean that many of those possibilities go to waste. For instance, concerns about the stability of content have led to an emphasis on the use of the PDF format, which makes any interaction with the content impossible. In other cases such as reviewing of texts, it has not been possible to achieve sufficient participation by specialist readers such as to be able to create a viable alternative to peer review.

     The reduction of costs involved in electronic publication has not always taken into account that journals are not free. The writing, editing, reviewing, translation, layout and dissemination require an effort that someone has to make, sometimes voluntarily way, but with a considerable cost that too frequently has to be borne by each of the participants in the process (to the detriment of other activities that these participants could be undertaking). This generosity should be acknowledged but it is not always taken into account, and this could limit the development of electronic journals.

Open Access

     The Budapest Open Access Initiative defines open access as the ability to obtain academic material with “free and unrestricted online availability.” Therefore, the emergence of Open Access in our context coincides with the availability on the Internet of this kind of material. It is therefore easy to comprehend that RELIEVE was the pioneer of Open Access within our context.

     It is difficult to exaggerate the changes that this new way of communicating implies for the scientific community. There are many legal initiatives that demand that research carried out with public funds be made available to other researchers through the Open Access journals.

     The growth of electronic journals has been spectacular in the last 20 years, fulfilling some of the expectations expressed some time ago (Aliaga y Suárez Rodríguez, 2002). The huge dissemination potential afforded by the Internet has changed the scope, reach, and impact of scientific journals. One of the advantages of publishing in open access electronic media is that it multiplies the effect of the reach and the impact of the articles, which is valuable for both authors and institutions. In our case, it allowed us to reach our first million readers in 2012, which is absolutely unthinkable for print journals. Furthermore, as a good example of the international reach the Net provides, Table 1 shows that Spain, though it is the country where RELIEVE was created and where it is published, is not the country with most readers of the journal.

TABLE 1 – Country of origin of visitors to the RELIEVE home page.

































Puerto Rico




United States
















Data from the Motigo Webstat, June 30th, 2014
RELIEVE’s achievements

     RELIEVE has always understood quality as a process of continuous improvement. This idea has allowed RELIEVE to achieve its goals and adapt to changes.

     The first achievement was to have appeared at a time when every step was an adventure, from getting access to the Internet to convincing institutions that magazines did not need to have physical copies to institutions. It is hard to acknowledge and thank sufficiently the pioneers that dared to take the first steps along this path but we wish to emphasise the work of Gregorio Rodriguez and the then President of AIDIPE, Javier Tejedor (Tejedor, 1995).

     The second achievement, which to many might have seemed incredible in that initial period, has been to keep the journal alive these 20 years. There are many people to whom we owe thanks for putting their shoulder to the wheel, from Jesus Suarez (who was the second director of the magazine) to all of the authors, reviewers, editors, and of course the readers and those who cited our articles. Without all their participation, the journal wouldn’t have survived these two exciting decades, and it would not have reached the present level of quality.

     I would like to give a special thanks to the authors whose articles have been rejected. Many of them had sent us interesting and rigorous articles. However, the need to seek a unique profile for RELIEVE has led us to focus on a specific type of article (by subject—only empirical studies, reviews or relevant advances—or by scope—generalizable and of interest to an international public) that made our magazine inadequate as a vehicle for some other work. We have always tried not to delay the process too much in these cases, so that the authors could have time to search for another journal more appropriate to the characteristics of their articles.

     We are aware that, obviously, every rejection can be frustrating, and even more so in the context in which we work in which this kind of rejection was not customary, and in which management creates a competitive and demanding environment. It may be that this frustration is sometimes expressed as criticism of the journal or its managers, but this is a risk we have to accept, since we are not prepared to reduce the quality of the magazine or blur its profile.

     Among RELIEVE’s achievements, we also must mention the recognition of the quality of the journal (Scopus, Latindex, ERIH, FECYT), and the admission to databases and relevant aggregators (Ebsco, Cengage, DOAJ, Redalyc etc.). These awards are due to the hard work and rigor of the journal, but more especially to the quality of the articles that we have been receiving and publishing. A range of different analyses (In-RECS, Zamora, Aguillo Ortega & Granadino, 2007) have shown that over the past years RELIEVE has achieved a high degree of impact and a broad reach.

     This decentralized dissemination implies greater access to articles published in RELIEVE, since they are not only available on the website (where it was easy to count them as it is a unique source) but also in many scientific information distributors. This new type of distribution adds complexity but, with any doubt, it means a greater reach and availability to a much larger international audience, as well as providing an indirect funding source for the magazine which though modest is not without importance.

    Although RELIEVE has always aimed to be bilingual, it has been just a few years since all our articles have been published in both Spanish and English. This was possible due to the positive response to the initiative on the part of our authors, and thanks to the great efforts of the editorial team, with the vital support of the University of Virginia (USA) Hispanic Studies Program, which provided trained and motivated staff who have supervised and/or translated all of the texts in English. This synergy between RELIEVE and the University of Virginia has not only led to the development of a professional environment to apply language skills, but also to a high degree of quality level and very substantial cost savings for both institutions. We want to show our gratitude to all participants in this collaboration, and especially to Mabel Richart and Leslie Atkinson.

     RELIEVE was the second journal created by the Interuniversity Association of Pedagogical Research (AIDIPE). To a large degree, it was the twin of the first journal, RIE . Journal of Educational Research (Bueno y Fernández-Cano, 2003), though it gave greater emphasis to educational evaluation. In this way, it developed an individual profile that allowed it to complement the Association’s other publishing project. While RIE had already achieved considerable prestige as a print journal with a national orientation (as regards authors, focus and readers) a notable number of theoretical proposals (Ariza y Quevedo-Blasco, 2013), with consolidated authors with extensive careers within the association, RELIEVE built its own profile by taking the following specific decisions:

  • the decision to be an electronic journal, with all the attendant implications from the point of view of management, reach and costs,

  • the decision to aim for an international focus from the start (Aliaga y Suárez, 2007), which has implications regarding origins, language and focus and the rejection of very locally focused research (which while interesting and necessary, has sufficient other distribution channels),

  • the choice of an emphasis on empirical research

  • the objective of a strong degree of editorial openness, which has attracted readers from all over the world and authors of different origins and background, including Spanish, but also Latin American, North American, European African and Asian.

     We believe therefore that both RIE and RELIEVE are journals that have shown a commitment to quality from different perspectives from the start and that they offer publishing alternatives with different focuses.

     All the development effort that has gone into RELIEVE has been characterised by enormous enthusiasm on the part of its creators, which has enabled us to overcome the precarious nature of its resources, since it is important to remember that RELIEVE did not receive any funding for more than half of its existence, and even now survives in difficult economic and organisational circumstances, and considerably more challenging conditions than other journals with which it has to compete.

Milestones in our history

Some of the major milestones that have marked the history of RELIEVE can be summarized as follows:

RELIEVE was assigned an ISSN on July 20th 1994, which can be viewed as the administrative birth of the journal.

The first number of the journal was distributed (by e-mail and by the prehistoric ‘gopher’ service) on January 15th 1995 : this could be described as its physical birth.

The management and editing of the journal became the responsibility of the University of Valencia in September 2001.

In 2003 we were included in the DOAJ Open Access journal platform and in 2004 e-magazines.

In 2004 we were included in the Latindex selective catalog

On September 8th 2007 we reached half million visits on the RELIEVE home page

In 2007 we formally subscribed to the Budapest Open Access Initiative, even though we already functioned as an Open Access initiative.

In 2008 we were granted the FECYT quality seal at the first opportunity (it was renewed in 2013, again at the first opportunity available).

In 2009 we were included in the well-regarded SCOPUS data base

In 2011 we were included in the European Reference Index for the Humanities (ERIH)

In August 2012: we reached one million visits to our home page.

Other milestone, or main action line goal, has been to reach the maximum possible readership. Although the aim was always to be an international journal, this aspect has received much more emphasis in recent years. Proof of this is that México, rather than Spain (where the journal is from), is the country that provides most visitors to RELIEVE. According to accumulated data from Motigo since 2003 (see Table 1) 75% of the readership of RELIEVE comes from other countries than Spain.

TABLE 2- Geographical areas of readers of RELIEVE





Central America



35,1 %

South America



34,6 %




26,0 %

USA and Canada



2,6 %




0,2 %




0,1 %




0,1 %

.net and .org



0,0 %




1,4 %




100,0 %

Data for visitors to RELIEVE homepage, according to Webstat Motigo. June 30 2014

     We are proud that RELIEVE serves the entire educational community, beyond borders or local limitations. It is an aspect that we wish to develop further, and this is why we have made great efforts in recent years to translate our articles so that all are available in both English and Spanish, the two most widespread languages. This allows us to reach an even wider audience.

     A very important aspect, given the times we are living in, is that RELIEVE has remained a completely free magazine for users, both for readers and authors. This achievement has to do with our understanding of "service to the community" and our aim to ensure maximum equality of opportunity for all (without discriminating against those with fewer economic resources).

     However, it is clear that publishing a scientific journal has its costs: time for review, editing, computing resources (servers, programming). If we have managed to keep the journal free it has been, in first place, thanks to the enthusiastic and selfless work of our many contributors (authors, reviewers, editors, etc.).

    We also should point out the financial support of the owner of RELIEVE, AIDIPE (Interuniversity Association for Pedagogical Research) and the University of Valencia, where it has been hosted in recent years (after previously using the computing resources of the CICA (Andalucía Computing Centre). Our sincere appreciation and recognition to all of them.

     We have been able to continue attracting the interest not only of readers but authors. Achieved thanks to RELIEVE’s fullfilling exigent quality criteria used in serveral countries to asses the scientific activity of researchers. Criteria and method that sometimes are not tranparent, public, stable and coherent enough (Galán & Zych, 2011).

Problems on the horizon

     These past 20 years of success and service to the international educational community service should not allow us to forget some of the problems that have arisen that affect scientific publications, in particular Open Access.

     On one hand we can point to the damage done to academic journals by the changes that took place at the end of the first decade of this century, and particularly the economic crisis. The reduction in the public resources available to develop or maintain tools for the evaluation of research has led to the disappearance of such interesting and promising projects as In-RECS, In-Rech, In.RECJ, DICE, or ERIH RESH. This is a serious problem the effects of which can probably only be seen in the medium and long term.

     The myopia of the managers of the institutions dedicated to the assessment of journals and policy makers has limited the opportunity to have tools that help us to understand the impact and quality of Social Sciences and Humanities journals, which due to their own features are focused on a mainly local environment (Galán, 2014), in contrast to the assessment tendency completely opposed to the one it’s being developed in our cultural environment (Delgado, 2010).

     The myopia of the management of public and private institutions managers has limited the opportunity to have tools that help us to understand the impact and quality of Social Sciences and Humanities journals, which due to their very characteristics are focused on a mainly local environment.

     Luckily, some national (FECYT) or regional (Latindex, SciELO, etc..) initiatives have been maintained, and these, though partially and with some problems, help to maintain a certain focus on our own community. This is fundamental in the Social Sciences and the Humanities, which is more focused on the local and the development of specific identities, as opposed to the dissemination of findings of international interest or implementation, as in the case of the ‘hard’ sciences.

     The decisions adopted by the managers in many countries have decanted most of the weight of journal evaluation to international commercial products (Web of Science, SCOPUS), which have a very high economic cost for the contributor (when compared to the journal evaluation tools with a more local orientation – In-RECS, DICE, RESH - which have disappeared due to the lack of a minimum of institutional support.

     In addition to the economic reasons, there are other scientific reasons which place into question the appropriateness of choices made by policy makers: commercial companies guide their decisions according to economic interest, and not necessarily scientific criteria, quality or actual impact. The decision by Thompson Reuters to substantially increase (and without sufficient justification) its coverage of different languages or scientific fields, not taken into account traditionally, has made some researchers doubt the degree of validity and objectivity of their products.

     A different problem has been emerging in the Open Access field. The commercial publishers have realized that the Open Access movement, thanks to the tools available, allows the researchers to develop their own communication media, so that much of the business, which is based on intermediation (both the supply of content and the demand come from researchers), could enter a crisis and suffer a drop in the number of subscriptions.

     The ingenious reaction has been, on the one hand, to turn the situation around and focus incomes on “payment for publishing” instead of (or as well as) “payment for subscribing” or “payment for reading”. The money does finally come from the same place, but this model makes it possible to show the hallmark of the Open Access. On the other hand, big corporations have set in motion a substantial apparatus (legal, economic, pressure etc.) to ensure that they are present in the most important databases, which ensures them considerable appeal to researchers whose incentives and professional careers depend on publishing in journals that are in those databases.

     Breaking out of this vicious circle is becoming harder and harder every time, and reactions such as those of the Nobel Prize-winner Schekman Nobel Price (2013) are more the privilege of someone who has already reached the summit of his profession than a serious option for an ordinary researcher. In other cases, the journal evaluation tools under development are suffering serious technical problems. Ayllón, Ruiz Pérez y Delgado López-Cózar (2013), have presented, grouped by disciplines, the impacts achieved by Spanish journals according to Google Scholar Metrics, as we can see in Image 1.

Figure 1- Ranking of Spanish educational journals according to Google Scholar Metrics

H index (2008-2012) of the Spanish scientific journals of education in Google Scholar Metrics.

Ayllón, Ruiz Pérez y Delgado López-Cózar (2013)

However, the rates and impacts calculated using Google Scholar have serious defects which limit their validity for use in evaluation. On one hand, they only collect citations that are automatically available online. On the other hand, in the case of academic journals there is a minimum of 20 published articles a year in order to appear in their listings.

Nevertheless, the number of published articles is an arbitrary criterion (why that number rather than any other?) and irrelevant: it says nothing about the quality of the journal or about its impact. As an example we can say that RELIEVE, which observes a very demanding, and therefore restrictive, policy in relation to what it publishes (with high rejection rates) publishes around 10 articles a year.

However, despite this (or perhaps due to this) it obtains excellent impact results, that could place it directly at the top of its category, even though RELIEVE has been excluded from the listings due to the ‘20 article’ criterion.

If we consider that RELIEVE could be included in this listing only by adding 10 or 12 articles of inferior quality each year, shows how arbitrary this criterion is. There are, therefore, a lot of problems to solve in relation to Google’s evaluation tools, although we think they provide a new, and worthy thrust to this field

 Figure 2- H index of RELIEVE as Google Scholar

Image taken June 30th, 2014 http://scholar.google.es/citations?user=m_KfyVsAAAAJ


Reviewing what we have done in the past two decades, the most important aspect for us is not so much to look back on what we have achieved, as to make the most of this anniversary to reflect and collect ourselves in order to improve further in the future.

We know that there is no excuse for giving up on the effort to improve in the world of scientific publishing. We are therefore about to enter a new period which we approach with renewed energy and desire to serve the world of education. But meanwhile, let’s celebrate this anniversary as it deserves: With your company.


Aliaga F. (1996). Enredados: aplicaciones y experiencias de Internet en España con interés educativo. Bordón, 48 (3), 355-361.

Aliaga, F. & Suárez Rodríguez, J (2002). Current tendencies in the publication of electronic journals: new stage in RELIEVE. RELIEVE, 8 (1). Consultado en http://www.uv.es/RELIEVE/v8n1/RELIEVEv8n1_0eng.htm.

Aliaga, F. & Suárez Rodriguez, J. (1995). Las redes de ordenadores: nuevas herramientas para la investigación educativa. (I) BITNET/EARN.  RELIEVE, 1(2). Consultado en http://www.uv.es/RELIEVE/v1/RELIEVEv1n2.htm

Aliaga, Francisco M. y Suárez-Rodríguez, Jesús M. (2007). Internationality of academic journals: A case study on RELIEVE. RELIEVE, 13 (1). http://www.uv.es/RELIEVE/v13n1/RELIEVEv13n1_0eng.htm

Aliaga. F. (2014). Calidad y evaluación de las revistas científicas de educación. Revista de Investigación Educativa, 32(1), 11-12.

Ariza, T. & Quevedo-Blasco, R. (2013). Análisis bibliométrico de la Revista de Investigación Educativa (2000-2012). Revista de Investigación Educativa, 31(1), pp. 31-52. Disponible en: http://revistas.um.es/rie/article/view/160321/142841

Ayllón, Juan Manuel; Ruiz Pérez, Rafael & Delgado López-Cózar, Emilio (2013). H Índice H de las revistas científicas españolas según Google Scholar Metrics (2008-2012). EC3 Reports,7. Granada, 18 de noviembre de 2013.

Bueno, A. & Fernández Cano, A. (2003). Análisis cienciométrico de la productividad en la Revista de Investigación Educativa (1983-2000). Revista de Investigación Educativa, 21(1), 507-532.

Delgado, Jorge Enrique (2010). Trends in the Publication of Refereed Journals in Spanish and Portuguese-Speaking Latin America?. Comparative & International Higher Education, 2 43-49

Galán, A. (2014). Evaluación de la producción científica en educación. ¿Una nueva crisis?.  Bordón, 66(2),  7-9. DOI: 10.13042/Bordon.2014.66200

Galán, A. y Zych, I. (2011). Análisis de los criterios de la Comisión Nacional Evaluadora de la Actividad Investigadora (CNEAI) para la concesión de los tramos de investigación en Educación. Bordón. Revista de pedagogía, 63 (2), 2011, pp. 117-139.

Gómez, E.L., Fuentes, J.L., & Luque, D. (2013). Análisis bibliométrico de las revistas españolas de educación incluidas en el Journal Citation Report. Producción científica y elementos controvertidos. Teoría de la Educación. Revista Interuniversitaria, 24(1), 183-217.

Hernández Pina, F. & Maquilón Sánchez, J. J. (2010). Indicadores de calidad de las revistas científicas y sistema de gestión editorial mediante OJS. Revista de investigación Educativa, 28(1), 13–29

Rodríguez Gómez, G. (1999). Revistas electrónicas: Cybernautas y/o papirófilos. Cambios en la comunicación científica. Revista de Investigación Educativa, 17 (2), pp. 491-494.

Rodríguez Gómez, Gregorio (2005). RELIEVE cumple 10 años, pero …¿cómo fue el primero?. RELIEVE, 11 (2). http://www.uv.es/RELIEVE/v11n2/RELIEVEv11n2_1.htm

Roes, H. (1994). Electronic journals: a survey of the literature and the net. Journal of Information Networking, 2(3), 169-186. http://cwis.kub.nl/~dbi/users/roes/articles/ej_join.htm

Schekman, R. (2013). Por qué revistas como ‘Nature’, ‘Science’ y ‘Cell’ hacen daño a la ciencia. El País, 12 de diciembre. Versión electrónica en http://sociedad.elpais.com/sociedad/2013/12/11/actualidad/1386798478_265291.html

Tejedor, F.J. (1995). Nuevas perspectivas en la comunicación científica. RELIEVE, 1 (0). Consultado en http://www.uv.es/RELIEVE/v1/RELIEVEv1n0.htm

Zamora, H; Aguillo, I; Ortega, J. L. & Granadino, B. (2007). Calidad formal, impacto y visibilidad de las revistas electrónicas universitarias españolas. El Profesional de la Información, 16 (1). Doi:10.3145/epi.2007.jan.02




Aliaga, Francisco M. (Francisco.Aliaga@uv.es). Professor in the Department of Research Methods and Diagnostic in Education at the Universidad de Valencia (Spain), the address of which is Avda. Blasco Ibañez, 30. 46010-Valencia (Spain).  His main areas of work are “Internet and Education”, “Research methodology” and “Research evaluation”. He was named Executive Editor of RELIEVE in September 2001 and was promoted to Director in June of 2009. Buscar otros artículos de este autor en Google Académico / Find other articles by this author in Scholar Google 



Reference /


 Aliaga, Francisco M. (2014). Twenty years of electronic publishing and open access: a pioneer reaches maturity.  RELIEVE, v. 20 (1), art. 0. DOI: 10.7203/relieve.20.1.3856

Title / Título

  Twenty years of electronic publishing and open access: a pioneer reaches maturity. [Veinte años de publicación electrónica y de acceso abierto: la madurez de una pionera].

Authors / Autores

  Aliaga, Francisco M.

Review / Revista

  RELIEVE  (Revista ELectrónica de Investigación y EValuación Educativa), v. 20 n. 1



Publication date /

Fecha de publicación

 2014 (Publication Date: 2014 June 30)

Abstract / Resumen

     RELIEVE reaches 20 years, a milestone in electronic publishing and the open access to educational research. We review some of the achievements and future prospects, as well as analyze some circumstances of journal evaluation processes.

    RELIEVE alcanza los 20 años, lo que supone un hito en la edición electrónica y en el acceso abierto a la investigación educativa. Revisamos algunos de los logros conseguidos y las perspectivas de futuro, a la vez que analizamos algunas circunstancias de los procesos de evaluación de revistas.    

Keywords / Descriptores

  Journal, journal articles, open access, publishing, electronic journals, evaluation, electronic publishing.  

  Revista, artículos de revista, acceso abierto, edición, revistas electrónicas, evaluación, edición electrónica. 

Institution / Institución

 University of Valencia (Spain).

Translator / Traducción   Sofia Kearney

Publication site / Dirección


Language / Idioma

Español & English version (Title, abstract and keywords in English & Spanish)


Volumen 20, n. 1


© Copyright, RELIEVE.  Reproduction and distribution of this article  is authorized if the content is no modified and its origin is indicated (RELIEVE Journal, volume, number and electronic address of the document).

© Copyright, RELIEVE.  Se autoriza la reproducción y distribución de este artículo siempre que no se modifique el contenido y se indique su origen (RELIEVE, volumen, número y dirección electrónica del documento).

[ ISSN: 1134-4032 ]

Revista ELectrónica de Investigación y EValuación Educativa

E-Journal  of  Educational  Research, Assessment  and  Evaluation




Statistics  Estadísticas